Monitoring your Laravel logs: the easy (and cheap) way
Discover how you can use Grafana to monitor your Laravel logs
In this post, we will see how to use a combination of easy-to-use and open source tools to monitor a Laravel application.
The main idea is to create a server (you can also use your computer for educational purpose) with Grafana installed, in which your production server sends the logs generated by Laravel. In this way, by accessing to Grafana through your browser, you will be able to create and use dashboards showing these data.
Terminology
Utilities server: the server where we will install Grafana
Production server: the server containing the Laravel application
Why two servers? Because if we use only one server, it will share its resources between what is already installed (application, web server…) with the monitoring system (and they are not so light).
Generally, it is not a good practice to have one server doing everything, because you are creating a single point of failure for your business. It’s important to keep things separated.
If you are only trying how this works, it is ok to use a local machine for everything.
Requirements
One server with Docker installed
Used tools
Create an aggregation system with Loki
Loki is a tool made by the Grafana team to aggregate logs from multiple sources.
Thanks to its semplicity, we will create a Docker container exposed on port 3100 where Grafana will be able to get the data.
So, let’s connect via SSH to the Utilities server and add the default configuration files. Now our goal is to have a simple configuration that works.
cd /var/lib
mkdir loki
cd loki
wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
Now, from the /var/lib/loki
directory, start the Docker container using the current directory for the loki configuration files. We will use the default port 3100.
docker run --name loki -d -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:3.0.0 -config.file=/mnt/config/loki-config.yaml
Now Loki is installed on the Utilities server and active on the port 3100.
Install and setup Grafana
If you don’t know Grafana, it is a tool that fetch data from a data source. For example, you can do a MySQL query and it will fetch the results automatically. You can use Grafana with many data sources, clearly including Loki. It has many features, including alerting, but for now we will limit to get the logs data.
The installation is really simple and must be made on the Utilities server. We will create a file docker-compose.yml
in the folder /var/lib/grafana
.
cd /var/lib
mkdir grafana
cd grafana
vim docker-compose.yml
Docker Compose file
Now let’s add the following code in the created file that tells to Docker which image to use and the exposed ports.
services:
grafana:
image: grafana/grafana-oss
container_name: grafana
restart: unless-stopped
ports:
- '3000:3000'
volumes:
- grafana-storage:/var/lib/grafana
volumes:
grafana-storage: {}
Finally, start the container using Docker Compose.
docker compose up -d
All done! Grafana is installed and you can now access http://[Utilities Server IP]:3000. Default credentials are admin/admin (change it!).
Add Loki as data source to Grafana
Now you can go to the Grafana sidebar and click to “Connections” → “Add new connection”. Select Loki as type and add http://172.17.0.1:3100
as the URL.
We are not using 127.0.0.1 because we are in a Docker container, so it doesn’t have anything on the 3100 port. Instead, 172.17.0.1 (on Unix systems, on Mac is host.docker.internal
) is the default bridge of the network, so it will connect to what is on the port 3100 of the Utilities server. Finally, click on “Save and test”. If everything is OK, you will receive “Data source successfully connected.”.
Send Laravel logs using Promtail
It’s the time to send Laravel logs using Promtail. Promtail is an agent that ships logs to a Loki instance. Log into your Production server and download Promtail.
wget https://github.com/grafana/loki/releases/download/v3.0.1/promtail-linux-amd64.zip
Unzip it and move to usr/local/bin
. In this way, it will be added to the system $PATH
.
unzip promtail-linux-amd64.zip
mv promtail-linux-amd64 /usr/local/bin/promtail
Configure Promtail
Then create the configuration file in which we tells the loki url and the path of the Laravel log file. The path is /etc/promtail-config.yaml
.
vim /etc/promtail-config.yaml
On the following content, edit the IP address
the host
and the __path__
variables.
server:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://[INSERT HERE THE UTILITIES IP]:3100/loki/api/v1/push
scrape_configs:
- job_name: laravel_logs
static_configs:
- targets:
- localhost
labels:
job: laravel_logs
host: [INSERT HERE THE DOMAIN]
__path__: [INSERT HERE THE PATH]/storage/logs/laravel.log
pipeline_stages:
- regex:
expression: '(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<env>\w+)\.(?P<level>\w+): (?P<message>.*)'
- labels:
level:
env:
- timestamp:
source: timestamp
format: '2006-01-02 15:04:05'
After this, we have configured everything and we want to check if it is working.
So, let’s start from the terminal Promtail using the config file just created.
promtail -config.file=/etc/promtail-config.yaml
If it is working, you will see some logs telling he is pushing data.
Run in the background
Now, probably you want to run it in the background, otherwise it will send logs only when you open via SSH your Production server and you run promtail.
To do this, let’s create a new service file for systemd.
vim /etc/systemd/system/promtail.service
Add the following code replacing the User value with the user having sufficient permissions (use root only for educational purposes on your machine!).
[Unit]
Description=Promtail service
After=network.target
[Service]
Type=simple
User=[INSERT HERE THE USER NAME WITH SUFFICIENT PERMISSIONS]
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail-config.yaml
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=promtail
[Install]
WantedBy=multi-user.target
So, now we have to reload systemd and start promtail.
systemctl daemon-reload
systemctl start promtail
systemctl enable promtail
Now we have promtail in background, sending data to Loki, that is scraped by Grafana!
Explore datas with Grafana
Make sure the laravel.log file is not empty, and go to the Grafana Dashboard. Go in the “Explore” page, and use the “Label filters” to select the job “laravel_logs”.
Using the button “Run query”, you will see a table called “Logs” containing the Laravel logs. If you want to filter only the errors, you could remove the “Line contains”, add an “Operation” of type “Label filters” → “Label filter expression”, and set the label as “level” and the value as “error”. In this way, if you re-run the query, you will obtain only the logs with level error.
Now you can go ahead building your dashboard containing these log files, and use Grafana to inspect them without accessing every time the server using SSH. This is also better if you have different Production servers, because it is sufficient to install Promtail on another machine to have it working, without touching anything on the Utilities server.