Microservices in Go

About Project · Report Bug · Request Feature

Table of Contents

• About project
• Project Structure
• Environment Variables
• Monitoring
• Getting Started
• Docker compose
• Kubernetes
• Usage
  • Front-end
  • Broker service
  • Authentication Service
  • Logger Service
  • Mail Service
  • Listener Service

About project

The "Microservices in Go" project is an open-source code repository that showcases how to create microservices using the Go programming language. The repository consists of several modules, each representing a separate microservice.

Currently, the following services have been partially implemented:

Service	Description
Front-End	Displays web page to test all services.
Broker	Routes requests to the appropriate backend service.
Authentication	A simple authentication service that uses Chi as the router and Postgres as the database.
Logging	Allows clients to write log entries to a MongoDB database.
Mail	Takes a JSON payload, converts into a formatted email, and sends it out.
Listener	Receives messages from RabbitMQ and acts upon them.

Project Structure

This project represents a multi service application consisting of a front-end, broker-service, authentication-service, logging-service, and mail-service. Each service has its own Dockerfile and Go module configuration. The directory structure is as follows:

.
├── project
│   ├── Makefile
│   ├── docker-compose.yml
│   └── src
│       ├── k8s
│       │   ├── auth-service.yml
│       │   ├─ ... other manifest
│       │   └── rabbit.yml
│       ├── grafana
│       │   └── provisioning
│       │       ├── dashboards
│       │       └── datasources
│       ├── prometheus
│       │   └── prometheus.yaml
│       └── script
│           └── init.sql
├── authentication-service
│   ├── authentication-service.dockerfile
│   ├── cmd
│   │   └── api
│   │       └── main.go / routes.go / handlers.go / helpers.go
│   ├── data
│   │   └── models.go
│   └── go.mod
├── broker-service
│   ├── broker-service.dockerfile
│   ├── cmd
│   │   └── api
│   │       └── main.go / routes.go / handlers.go / helpers.go
│   ├── data
│   │   └── models.go
│   ├── event
│   │   ├── data
│   │   └── logger.go / event.go / emitter.go / consumer.go
│   ├── logs
│   │   └── logs_grpc.pb.go / logs.proto / logs.pb.go
│   └── go.mod
├── front-end
│   ├── cmd
│   │   └── web
│   │       ├── main.go
│   │       └── templates
│   │           └── ... 
│   ├── frontend-service.dockerfile
│   └── go.mod
├── listener-service
│   ├── event
│   │   └── logger.go / consumer.go / event.go
│   ├── main.go
│   ├── listener-service.dockerfile
│   └── go.mod
├── logging-service
│   ├── cmd
│   │   └── api
│   │       └── main.go / routes.go / handlers.go / helpers.go / rpc.go
│   ├── data
│   │   └── models.go
│   ├── logs
│   │   └── logs_grpc.pb.go / logs.proto / logs.pb.go
│   ├── logger-service.dockerfile
│   └── go.mod
└── mail-service
    ├── cmd
    │   └── api
    │       └── main.go / routes.go / handlers.go / helpers.go / mailer.go 
    ├── go.mod
    ├── mail-service.dockerfile
    └── templates
        └── ...

(back to the Table of content)

Environment Variables

Authentication Service

The authentication service uses the DSN environment variable that contains the connection string for the PostgreSQL database, including host, port, user, password, database name, SSL mode, timezone, and connection timeout.

Example:

host=postgres port=5432 user=postgres password=password dbname=users sslmode=disable timezone=UTC connect_timeout=5

Mailer Service

The mailer service uses the following environment variables:

Variable	Description	Example
MAIL_DOMAIN	The domain name of the email server.	localhost
MAIL_HOST	The hostname of the email server.	mailhog
MAIL_PORT	The port number of the email server.	1025
MAIL_ENCRYPTION	The encryption method for email.	none
MAIL_USERNAME	The username for the email server.	""
MAIL_PASSWORD	The password for the email server.	""
FROM_NAME	The display name for the sender of the emails.	"Konstantin Evo"
FROM_ADDRESS	The email address of the sender.	[email protected]

PostgreSQL

The PostgreSQL service uses the following environment variables:

Variable	Description	Example
POSTGRES_USER	The username for the PostgreSQL database.	postgres
POSTGRES_PASSWORD	The password for the PostgreSQL database.	password
POSTGRES_DB	The name of the PostgreSQL database.	users

MongoDB

The MongoDB service uses the following environment variables:

Variable	Description	Example
MONGO_INITDB_DATABASE	The name of the MongoDB database.	logs
MONGO_INITDB_ROOT_USERNAME	The root username for the MongoDB database.	admin
MONGO_INITDB_ROOT_PASSWORD	The root password for the MongoDB database.	password

(back to the Table of content)

Monitoring

Project includes a monitoring setup using Prometheus and Grafana, providing real-time insights and performance metrics.

The Prometheus data source is used for all panels, ensuring high accuracy and reliability of the metrics. The dashboard is set to refresh every 5 seconds to provide real-time updates.

Dashboard Overview

The monitoring dashboard includes several key panels, each visualizing critical metrics to help you track the health and performance of your application:

1. Application Response Time

   • Description: Measures the average time it takes for the application to respond to a request.
   • Metric: sum(rate(gin_request_duration_sum[$range])) / sum(rate(gin_request_duration_count[$range]))
   • Visualization: Line chart showing response time over the selected range.

2. Requests Per Second (RPS)

   • Description: Indicates the rate at which your application receives and processes incoming requests.
   • Metric: sum(rate(gin_request_total[$range]))
   • Visualization: Line chart displaying the number of requests per second over time.

3. PV Rate

   • Description: Shows the application request rate.
   • Metric: rate(gin_request_total[$range])
   • Visualization: Gauge displaying the rate of requests.

4. Method Distribution

   • Description: Breakdown of requests by method (GET, POST, etc.).
   • Metric: sum by(method) (gin_uri_request_total)
   • Visualization: Pie chart depicting the proportion of each method type.

5. Response Code Distribution

   • Description: Breakdown of HTTP response codes.
   • Metric: sum by(code, instance) (gin_uri_request_total)
   • Visualization: Pie chart showing the distribution of response codes (e.g., 200, 404).

6. Traffic In-Out

   • Description: Measures the traffic in and out of the application.
   • Metrics:
     • Inbound: rate(gin_request_body_total[$range])
     • Outbound: rate(gin_response_body_total[$range])
   • Visualization: Line chart showing inbound and outbound traffic rates.

7. Slow Request Detection

   • Description: Tracks slow requests to identify potential performance bottlenecks.
   • Metric: sum by(uri, instance) (gin_slow_request_total)
   • Visualization: Bar gauge indicating the number of slow requests.

Getting Started

To get started, clone this repository and navigate to the project directory:

git clone https://github.com/konstantin-evo/microservices-in-go.git
cd microservices-in-go

You can run the application locally using Docker compose or on a Kubernetes cluster.

Docker compose

Prerequisites:

• Go 1.18
• Docker
• MongoDB Compass (only to see data in MongoDB database via GUI)

Start the project using docker-compose

cd ./project && docker-compose up -d

Endpoints:

Service	URL	Credentials	Description
Front-end	http://localhost	-	The main application interface
Mail server	http://localhost:8025/	-	MailHog UI to view captured emails
Database	http://localhost:8089/	POSTGRES_USER: postgres, POSTGRES_PASSWORD: password, POSTGRES_DB: users	Access to the users database
RabbitMQ	http://localhost:15672/	guest:guest	RabbitMQ management console
Prometheus	http://localhost:9090/	-	Prometheus monitoring and alerting toolkit
Grafana	http://localhost:3000/	admin:admin	Grafana analytics and monitoring dashboard

Note: To view the data in the "logs" database (used in Logging Service), it is necessary to install MongoDB Compass.

You can find more information about connection to PostgreSQL and MongoDB in the Environment Variables section

(back to the Table of content)

Kubernetes

Prerequisites:

Before getting started, make sure you have the following installed on your machine:

1. Docker
2. kubectl
3. A Kubernetes cluster such as Minikube or a cloud provider such as GKE or AKS

Also, ensure that your Kubernetes cluster is properly configured and that you have appropriate permissions to deploy applications to it. If you are using a cloud provider, you may need to configure additional settings such as network policies or load balancers.

The YAML files in the k8s directory define the deployments, services, and ingress resources needed to run the application on Kubernetes.

To run project:

1. Apply all the YAML files in the directory by running the kubectl apply -f . command
2. Wait for all the services to start running.

Once all the services are running, you should be able to access the application by navigating to the URL http://front-end.info

(back to the Table of content)

Usage

Front-end

This project is a front-end service that interacts with microservices in Go.

The application interface consists of buttons to test each microservice and fields to input authentication details.

To test a microservice, click on the corresponding button. The application will send a request to the microservice and display the response in the "Received" section. The request payload will be displayed in the "Sent" section.

If an error occurs, the error message will be displayed in the "Logs" section.

(back to the Table of content)

Broker Service

The broker service serves as a proxy between clients and various backend services. It is responsible for routing requests to the appropriate backend service.

The broker service utilizes RabbitMQ to handle messaging with logger service and also supports gRPC communication for specific interactions with the Logger Service.

Endpoints

The broker service provides two endpoints:

• /
• /handle

GET / is a simple endpoint that returns a response indicating that the broker service is up and running.

Response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "error": false,
  "message": "Hit the broker"
}

POST /handle

This endpoint is used to handle requests from clients. The broker service expects a JSON payload with an action field indicating the type of action to perform, and an optional auth field containing authentication information.

The request and response depend on the action field. The example for authentication:

{
  "action": "auth",
  "auth": {
    "email": "[email protected]",
    "password": "password123"
  }
}

If the action field is not recognized, the response will contain an error message:

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "error": true,
  "message": "unknown action"
}

For the Logger Service, the Broker Service supports both RabbitMQ messaging and RPC communication. The Broker Service uses the logEvent function for RabbitMQ messaging and the logItemViaRPC function for RPC communication.

(back to the Table of content)

Authentication Service

This is a simple authentication service built in Go, which uses Chi as the router and Postgres as the database.

The "users" database containing user data can be accessed at http://localhost:8089/

To connect to the database, you need to provide the following information:

• server: postgres
• username: postgres
• password: password
• database: users

Endpoints

The following endpoints are available:

POST /authenticate - authenticate a user.

The request payload should contain the following fields: email and password. If the credentials are valid, the endpoint will return a JSON response containing the user object and a success message. If the credentials are invalid, it will return an error message.

Example request:

POST /authenticate HTTP/1.1
Host: localhost:80
Content-Type: application/json

{
  "email": "[email protected]",
  "password": "password123"
}

Example response:

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "error": false,
  "message": "Logged in user [email protected]",
  "data": {
    "id": 1,
    "email": "[email protected]",
    "first_name": "John",
    "last_name": "Doe",
    "active": 1,
    "created_at": "2022-10-15T14:35:00Z",
    "updated_at": "2022-10-15T14:35:00Z"
  }
}

If the email and password are invalid, the server will respond with an error message:

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "error": true,
  "message": "invalid credentials"
}

Structure

The code is structured as follows:

• cmd/api/main.go - the main entry point for the application.
• cmd/api/routes.go - the routing and middleware configuration for the application.
• cmd/api/handlers.go - the request handlers for the endpoints.
• cmd/api/helpers.go - some helper functions for parsing JSON, writing JSON responses, and handling errors.
• data/models.go - the database models for the application.
• authentication-service.dockerfile - the Dockerfile for the application.

(back to the Table of content)

Logger Service

The Logger Service is a versatile logging system that allows clients to write log entries to a MongoDB database. It supports two primary modes of communication: REST (HTTP) and RPC.

Different services can use the Logger Service in different ways:

1. The Auth Service communicates with the Logger Service via HTTP to send a POST request when a user is successfully authenticated.
2. On the other hand, the Broker Service utilizes RPC communication to interact with the Logger Service remotely, by calling the LogInfo method of the RPCServer struct.

In summary, the Logger Service provides flexibility in communication methods, enabling various services to interact with it seamlessly, whether it be through REST (HTTP) or RPC.

Endpoints

The POST /log endpoint is used to write log entries to the MongoDB database. It expects a JSON payload that contains the name and data fields.

Request Example:

POST /log HTTP/1.1
Content-Type: application/json

{
  "name": "info",
  "data": "This is an info message."
}

Response Example:

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "error": false,
  "message": "The message is successfully logged in the service.",
  "data": null
}

Structure

The code is structured as follows:

• cmd/api/main.go - the main entry point for the application.
• cmd/api/routes.go - the routing and middleware configuration for the application.
• cmd/api/rpc.go - the RPC server implementation and related functions.
• cmd/api/handlers.go - the request handlers for the endpoints.
• cmd/api/helpers.go - some helper functions for parsing JSON, writing JSON responses, and handling errors.
• data/models.go - the database models for the application.
• logger-service.dockerfile - the Dockerfile for the application.

(back to the Table of content)

Mail Service

This is a simple mail service API, which allows you to send emails using the SMTP protocol.

Features:

1. Send emails with attachments
2. Supports HTML and plain text email templates
3. Environmental variable configuration for mail settings
4. Containerization with Docker

Demo:

Endpoints

The following endpoints are available:

1. POST /send: Send an email
2. GET /ping: Health check endpoint to verify the service is running

To send an email, make a POST request to /send with the following JSON payload:

{
  "from": "[email protected]",
  "to": "[email protected]",
  "subject": "Hello, World!",
  "message": "This is a test email."
}

Response Payload

{
  "error": false,
  "message": "The message successfully sent to [email protected]"
}

To verify the service is running, make a GET request to /ping. If the service is running correctly, it will return a 200 OK status.

The code is organized into the following files:

Structure

The code is structured as follows:

• handlers.go: Contains the HTTP handler functions for the API endpoints
• routes.go: Configures the API routes and CORS settings
• mailer.go: Contains the email sending logic and email template rendering
• helpers.go: Contains helper functions for JSON input/output and error handling
• main.go: Initializes the mail service configuration and starts the HTTP server
• go.mod and go.sum: Go module dependency management files
• mail-service.dockerfile: Dockerfile to containerize the mail service
• templates/mail.plain.gohtml: Plain text email template
• templates/mail.html.gohtml: HTML email template

(back to the Table of content)

Listener Service

This is a simple application to listen for and consume messages from RabbitMQ based on specific topics. The application is written in Go and uses the RabbitMQ Go client library for communication with RabbitMQ.

Features:

1. Connect to a RabbitMQ server.
2. Listen for messages with specific topics.
3. Forward the consumed messages to a log service.

Structure

The code is structured as follows:

• main.go: Initializes the configuration and connects to RabbitMQ, sets up the consumer, and starts listening for messages.
• event/consumer.go: Contains the Consumer struct and methods for setting up and consuming messages from RabbitMQ.
• event/exchange.go: Contains the function for declaring the topic exchange.
• event/queue.go: Contains the function for declaring a random queue and binding it to the topic exchange.
• event/log_event.go: Contains the function for sending the consumed messages to the log service.
• go.mod and go.sum: Go module dependency management files.

Configuration

The application uses a simple configuration struct to set up the connection to RabbitMQ and the log service URL:

app := config{
  RabbitMqURL:   "amqp://guest:guest@rabbitmq",
  LogServiceURL: "http://logger-service/log",
  Topics:        []string{"log.INFO", "log.WARNING", "log.ERROR"},
}

You can modify these values according to your environment.

(back to the Table of content)