Thinh Dang

Experienced Fintech Software Engineer Driving High-Performance Solutions

As a software developer, I have always been interested in building complex applications that involve multiple microservices, asynchronous tasks, long-running workflows, and distributed transactions. However, I also faced many challenges in developing such applications, such as dealing with error handling, retries, timeouts, compensation, and state management. I also had to balance between consistency and availability, as well as scalability and reliability of my applications.

That’s why I was curious when I discovered Temporal, a platform for workflow orchestration and management. Temporal claimed to simplify the development of reliable and scalable applications by providing a simple programming model for writing workflows and activities using familiar languages and tools. Temporal also claimed to provide reliable and durable execution for services and applications in the presence of any failure.

I decided to give Temporal a try and see if it could live up to its promises. In this blog post, I will share my experience of learning how to use Temporal effectively and how it helped me build reliable and scalable applications.

What is Temporal?

Temporal is a distributed, scalable, durable, and highly available orchestration engine that executes asynchronous long-running business logic in a scalable and resilient way. Temporal provides reliability primitives such as seamless state tracking, automatic retries, timeouts, rollbacks due to process failures, etc.

Temporal consists of two main components: the Temporal Server and the Temporal SDKs.

  • The Temporal Server is a scalable service that runs on your infrastructure or on the cloud. It tracks the progress of your workflow executions and coordinates the execution of activities across multiple workers. The Temporal Server also provides features such as task management, error handling, retries, compensation, signals, queries, etc.

  • The Temporal SDKs are libraries that allow you to write workflows and activities using your preferred programming language. Temporal supports Go and Java SDKs natively, and there are community-supported SDKs for other languages such as Python and Ruby. The Temporal SDKs communicate with the Temporal Server via gRPC and provide abstractions for writing workflows and activities.

The core concepts of Temporal are workflows and activities.

  • A workflow is a function that orchestrates the execution of activities across multiple workers. A workflow can be written in any programming language that is supported by the Temporal SDKs. A workflow can also be composed of other workflows or sub-workflows. A workflow can run for minutes, hours, days or even years without losing its state.

  • An activity is a function that performs a single well-defined action such as calling an API, sending an email or processing a payment. An activity can be written in any programming language that is supported by the Temporal SDKs. An activity can also be implemented by an external service or system that can be invoked via HTTP or gRPC.

  • A worker is a process that hosts workflows and activities. A worker can run on any machine or container that can communicate with the Temporal Server. A worker can register itself with one or more task queues that specify which workflows or activities it can execute.

  • A task is a unit of work that is sent from the Temporal Server to a worker via a task queue. A task can be either a workflow task or an activity task. A workflow task contains the state of a workflow execution and instructs the worker to execute the next step of the workflow logic. An activity task contains the input parameters of an activity function and instructs the worker to execute it.

  • A signal is a way to send external events or data to a running workflow instance. A signal can be sent from another workflow instance or from an external service or system via the Temporal SDKs or CLI. A signal can trigger custom logic in the workflow or update its state.

  • A query is a way to get information about the state of a running workflow instance. A query can be executed from another workflow instance or from an external service or system via the Temporal SDKs or CLI. A query can return any data that is accessible by the workflow code.

How I used Temporal effectively

To learn how to use Temporal effectively, I decided to follow some tutorials and examples from the official documentation (https://docs.temporal.io/docs/go/hello-world) and from some blog posts (https://dev.to/ifedayo/building-more-reliable-applications-with-temporal-44d0). I also watched some videos (https://www.youtube.com/watch?v=f-18XztyN6c).

I started by setting up a local development environment for developing Temporal applications using the Go programming language. I installed Go and Docker on my machine and verified that they were working properly. I also installed the Temporal CLI (tctl) which is a command-line tool that allows me to interact with the Temporal Server and perform various tasks such as creating namespaces, starting workflows, sending signals, executing queries, etc.

I then downloaded the Temporal Server docker image from Docker Hub (https://hub.docker.com/r/temporalio/auto-setup) and ran it on my machine using the following command:

docker run -d --name temporal --network host temporalio/auto-setup:1.12.2

This command started a Temporal Server instance on my machine that listened on port 7233 for gRPC requests from workers and clients. It also created a default namespace called “default” that I could use for my applications.

I also downloaded the Temporal Web UI docker image from Docker Hub (https://hub.docker.com/r/temporalio/web) and ran it on my machine using the following command:

docker run -d --name temporal-web --network host -e TEMPORAL_GRPC_ENDPOINT=localhost:7233 temporalio/web:1.12.2

This command started a Temporal Web UI instance on my machine that listened on port 8088 for HTTP requests from browsers. It also connected to the Temporal Server instance via gRPC and provided a graphical interface for monitoring and troubleshooting my workflow executions.

I then created a new directory called “hello-world” on my machine and initialized a Go module inside it using the following commands:

mkdir hello-world
cd hello-world
go mod init hello-world

I also added the Temporal Go SDK as a dependency to my Go module using the following command:

go get go.temporal.io/sdk@v1.12.2

This command downloaded the Temporal Go SDK library to my Go module cache and updated my go.mod file with the required version.

I then created three Go files inside my hello-world directory: main.go, starter.go, and worker.go.

The main.go file contained the main function that ran both the starter and worker components of my application. It looked like this:

package main

import (
	"log"
)

func main() {
	// Start a worker that hosts both Workflow and Activity implementations
	go startWorker()

	// Start a starter that starts Workflow Executions
	startStarter()
}

The starter.go file contained the starter component that started workflow executions using the Temporal client. It looked like this:

package main

import (
	"context"
	"log"

	"go.temporal.io/sdk/client"
)

const (
	TaskQueue = "hello-world-task-queue"
)

func startStarter() {
	// Create the client object just once per process
	c, err := client.NewClient(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	// This value is used to identify this run of the workflow
	workflowID := "hello-world-workflow"

	// Start a workflow execution
	options := client.StartWorkflowOptions{
		ID:        workflowID,
		TaskQueue: TaskQueue,
	}

	we, err := c.ExecuteWorkflow(context.Background(), options, helloWorldWorkflow)
	if err != nil {
		log.Fatalln("Unable to execute workflow", err)
	} else {
		log.Println("Started workflow", "WorkflowID", we.GetID(), "RunID", we.GetRunID())
	}
}

The worker.go file contained the worker component that hosted both workflow and activity implementations. It looked like this:

package main

import (
	"context"
	"fmt"
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"
	"go.temporal.io/sdk/workflow"
)

const (
	TaskQueue = "hello-world-task-queue"
)

func startWorker() {
	// Create the client object just once per process
	c, err := client.NewClient(client.Options{}) if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	// Create a worker that listens on a task queue
	w := worker.New(c, TaskQueue, worker.Options{})

	// Register workflow and activity implementations
	w.RegisterWorkflow(helloWorldWorkflow)
	w.RegisterActivity(helloWorldActivity)

	// Start listening to the task queue
	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}

The helloWorldWorkflow function was a simple workflow that executed an activity that printed “Hello World!” to the console. It looked like this:

func helloWorldWorkflow(ctx workflow.Context) error {
	logger := workflow.GetLogger(ctx)
	logger.Info("Hello World Workflow started")

	var result string
	err := workflow.ExecuteActivity(ctx, helloWorldActivity).Get(ctx, &result)
	if err != nil {
		logger.Error("Activity failed.", "Error", err)
		return err
	}

	logger.Info("Hello World Workflow completed.")
	return nil
}

The helloWorldActivity function was a simple activity that returned “World” as a result. It looked like this:

func helloWorldActivity(ctx context.Context) (string, error) {
	fmt.Println("Hello World!")
	return "World", nil
}

I then ran my application using the following command:

go run main.go

This command started both the starter and worker components of my application. The starter component started a workflow execution with the ID “hello-world-workflow” and sent it to the Temporal Server via gRPC. The Temporal Server then assigned a workflow task to one of the workers that registered with the task queue “hello-world-task-queue”. The worker received the workflow task and executed the next step of the workflow logic, which was to execute an activity with the input parameters of the helloWorldActivity function. The worker then sent an activity task to another worker that registered with the same task queue. The worker received the activity task and executed the helloWorldActivity function, which printed “Hello World!” to the console and returned “World” as a result. The worker then reported the activity result back to the Temporal Server via gRPC. The Temporal Server then assigned another workflow task to one of the workers to continue the workflow execution. The worker received the workflow task and completed the workflow execution by logging “Hello World Workflow completed.” to the console.

I then opened my browser and navigated to http://localhost:8088/ to access the Temporal Web UI. I saw that my workflow execution was listed in the dashboard with its ID, status, start time, end time, etc. I clicked on my workflow execution and saw more details about it, such as its history, summary, stack trace, etc. I also saw that I could perform actions on my workflow execution such as terminate it, signal it or query it.

I was impressed by how easy it was to write and run a simple workflow using Temporal. I decided to explore more features of Temporal and see how it could help me with more complex scenarios.

How Temporal helped me with more complex scenarios

After learning how to write and run a simple workflow using Temporal, I wanted to see how Temporal could help me with more complex scenarios that involved multiple microservices, asynchronous tasks, long-running workflows, and distributed transactions.

I decided to try out some of the examples and tutorials from the official documentation (https://github.com/temporalio/samples-go) and from some blog posts (https://www.temporal.io/blog).Temporal could be used for various use cases such as business transactions, business process applications, infrastructure management, etc.

Some of the scenarios that I tried out were:

  • Money transfer: This scenario involved transferring money from one account to another using two microservices: one for withdrawing money from an account and another for depositing money into an account. I used Temporal to orchestrate this transaction as a workflow that executed two activities: one for calling the withdraw service and another for calling the deposit service. I also used Temporal’s features such as retries, timeouts, compensation, and signals to handle errors, failures, and cancellations. I learned how Temporal could ensure the consistency and reliability of this transaction even in the presence of network issues or service outages.

  • Cron workflow: This scenario involved running a periodic task that performed some action such as sending an email or processing a batch job. I used Temporal to implement this task as a cron workflow that executed an activity on a specified schedule. I learned how Temporal could provide durability and scalability for this task even if the worker or the machine that ran it crashed or restarted.

  • Greeting workflow: This scenario involved greeting a user based on their preferences and location. I used Temporal to implement this task as a workflow that executed three activities: one for getting the user’s name, another for getting the user’s preferred language, and another for getting the user’s timezone. I learned how Temporal could provide flexibility and extensibility for this task by allowing me to add more activities or change the workflow logic without affecting the existing workflow instances.

  • Expense report workflow: This scenario involved processing an expense report submitted by an employee and approving or rejecting it by a manager. I used Temporal to implement this task as a workflow that executed several activities: one for validating the expense report, another for sending an email notification to the manager, another for waiting for the manager’s approval or rejection, and another for updating the database with the final status. I learned how Temporal could provide long-running and stateful execution for this task by allowing me to query the workflow state at any point or send signals to change the workflow behavior.

  • Video processing workflow: This scenario involved processing a video uploaded by a user and applying some transformations such as resizing, cropping, or adding filters. I used Temporal to implement this task as a workflow that executed several activities: one for downloading the video from a storage service, another for applying the transformations using a third-party service, another for uploading the processed video back to the storage service, and another for sending an email notification to the user. I learned how Temporal could provide parallel and asynchronous execution for this task by allowing me to execute multiple activities concurrently or in any order.

I was amazed by how Temporal could help me with these complex scenarios and how it could simplify the development of reliable and scalable applications. I also appreciated how Temporal provided me with visibility and control over my workflow executions using the Temporal Web UI and CLI. I could monitor the status, history, and details of each workflow execution, as well as perform actions such as terminate, signal, or query workflows.

Conclusion

In this blog post, I shared my experience of learning how to use Temporal effectively and how it helped me build reliable and scalable applications. I explained what Temporal is and what are its main components and concepts. I also showed how I used Temporal to write and run simple and complex workflows using various features and tools.

I hope you enjoyed reading this blog post and found it useful. If you are interested in learning more about Temporal or trying it out yourself, you can visit their website (https://www.temporal.io/) or their GitHub repository (https://github.com/temporalio/temporal).

Thank you for reading!

You may also enjoy