Monitoring your Laravel logs: the easy (and cheap) way

Discover how you can use Grafana to monitor your Laravel logs

Sep 04, 2024

In this post, we will see how to use a combination of easy-to-use and open source tools to monitor a Laravel application.

The main idea is to create a server (you can also use your computer for educational purpose) with Grafana installed, in which your production server sends the logs generated by Laravel. In this way, by accessing to Grafana through your browser, you will be able to create and use dashboards showing these data.

Terminology

Utilities server: the server where we will install Grafana
Production server: the server containing the Laravel application

Why two servers? Because if we use only one server, it will share its resources between what is already installed (application, web server…) with the monitoring system (and they are not so light).

Generally, it is not a good practice to have one server doing everything, because you are creating a single point of failure for your business. It’s important to keep things separated.

If you are only trying how this works, it is ok to use a local machine for everything.

Requirements

One server with Docker installed

Used tools

Create an aggregation system with Loki

Loki is a tool made by the Grafana team to aggregate logs from multiple sources.
Thanks to its semplicity, we will create a Docker container exposed on port 3100 where Grafana will be able to get the data.

So, let’s connect via SSH to the Utilities server and add the default configuration files. Now our goal is to have a simple configuration that works.

cd /var/lib
mkdir loki
cd loki
wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

Now, from the /var/lib/loki directory, start the Docker container using the current directory for the loki configuration files. We will use the default port 3100.

docker run --name loki -d -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:3.0.0 -config.file=/mnt/config/loki-config.yaml

Now Loki is installed on the Utilities server and active on the port 3100.

Install and setup Grafana

If you don’t know Grafana, it is a tool that fetch data from a data source. For example, you can do a MySQL query and it will fetch the results automatically. You can use Grafana with many data sources, clearly including Loki. It has many features, including alerting, but for now we will limit to get the logs data.

The installation is really simple and must be made on the Utilities server. We will create a file docker-compose.yml in the folder /var/lib/grafana.

cd /var/lib
mkdir grafana
cd grafana
vim docker-compose.yml

Docker Compose file

Now let’s add the following code in the created file that tells to Docker which image to use and the exposed ports.

services:
  grafana:
    image: grafana/grafana-oss
    container_name: grafana
    restart: unless-stopped
    ports:
      - '3000:3000'
    volumes:
      - grafana-storage:/var/lib/grafana
volumes:
  grafana-storage: {}

Finally, start the container using Docker Compose.

docker compose up -d

All done! Grafana is installed and you can now access http://[Utilities Server IP]:3000. Default credentials are admin/admin (change it!).

Add Loki as data source to Grafana

Now you can go to the Grafana sidebar and click to “Connections” → “Add new connection”. Select Loki as type and add http://172.17.0.1:3100 as the URL.

We are not using 127.0.0.1 because we are in a Docker container, so it doesn’t have anything on the 3100 port. Instead, 172.17.0.1 (on Unix systems, on Mac is host.docker.internal) is the default bridge of the network, so it will connect to what is on the port 3100 of the Utilities server. Finally, click on “Save and test”. If everything is OK, you will receive “Data source successfully connected.”.

Send Laravel logs using Promtail

It’s the time to send Laravel logs using Promtail. Promtail is an agent that ships logs to a Loki instance. Log into your Production server and download Promtail.

wget https://github.com/grafana/loki/releases/download/v3.0.1/promtail-linux-amd64.zip

Unzip it and move to usr/local/bin. In this way, it will be added to the system $PATH.

unzip promtail-linux-amd64.zip
mv promtail-linux-amd64 /usr/local/bin/promtail

Configure Promtail

Then create the configuration file in which we tells the loki url and the path of the Laravel log file. The path is /etc/promtail-config.yaml.

vim /etc/promtail-config.yaml

On the following content, edit the IP address the host and the __path__ variables.

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://[INSERT HERE THE UTILITIES IP]:3100/loki/api/v1/push

scrape_configs:
  - job_name: laravel_logs
    static_configs:
    - targets:
        - localhost
      labels:
        job: laravel_logs
        host: [INSERT HERE THE DOMAIN]
        __path__: [INSERT HERE THE PATH]/storage/logs/laravel.log
    pipeline_stages:
    - regex:
        expression: '(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<env>\w+)\.(?P<level>\w+): (?P<message>.*)'
    - labels:
        level:
        env:
    - timestamp:
        source: timestamp
        format: '2006-01-02 15:04:05'

After this, we have configured everything and we want to check if it is working.
So, let’s start from the terminal Promtail using the config file just created.

promtail -config.file=/etc/promtail-config.yaml

If it is working, you will see some logs telling he is pushing data.

Run in the background

Now, probably you want to run it in the background, otherwise it will send logs only when you open via SSH your Production server and you run promtail.

To do this, let’s create a new service file for systemd.

vim /etc/systemd/system/promtail.service

Add the following code replacing the User value with the user having sufficient permissions (use root only for educational purposes on your machine!).

[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=[INSERT HERE THE USER NAME WITH SUFFICIENT PERMISSIONS]
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail-config.yaml
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=promtail

[Install]
WantedBy=multi-user.target

So, now we have to reload systemd and start promtail.

systemctl daemon-reload
systemctl start promtail
systemctl enable promtail

Now we have promtail in background, sending data to Loki, that is scraped by Grafana!

Explore datas with Grafana

Make sure the laravel.log file is not empty, and go to the Grafana Dashboard. Go in the “Explore” page, and use the “Label filters” to select the job “laravel_logs”.

Using the button “Run query”, you will see a table called “Logs” containing the Laravel logs. If you want to filter only the errors, you could remove the “Line contains”, add an “Operation” of type “Label filters” → “Label filter expression”, and set the label as “level” and the value as “error”. In this way, if you re-run the query, you will obtain only the logs with level error.

Now you can go ahead building your dashboard containing these log files, and use Grafana to inspect them without accessing every time the server using SSH. This is also better if you have different Production servers, because it is sufficient to install Promtail on another machine to have it working, without touching anything on the Utilities server.

tchury

Discussion about this post