With the recent announcement of Observability and Management platform I thought it’s good to try out how can we fetch logs from provisioned instances in OCI.
In this post I’ll be making sure the instance I’m using has the Monitoring Agent required installed. Also I’ll make sure I have Log Group, Logs and Agent Configuration defined on OCI side.
Finally I’ll verify we are seeing the Custom Logs on OCI Console.
Setting up the instance
While provisioning new instance we need to make sure we are deploying the instance with Monitoring Enabled. If this is done then on the instance we don’t need to do anything else.
If you have other instances which are deployed without monitoring, no problem! You can simply download RPM from Object Storage as mentioned in the documentation, same applies for other supported Operating Systems.
But now when checking on the instance if the agent is running I see following:
[opc@compute-public1 ~]$ systemctl status unified-monitoring-agent.service
Unit unified-monitoring-agent.service could not be found.
Based on documentation following pre-requisites should fill the need:
- Monitoring must be enabled for the instance.
- The instance must have one of the supported operating systems:
- Oracle Linux 7/8
- CentOS 7 (though the image for 8 has not been released yet, but it is also supported)
I’m using OL7 and verified monitoring is enabled. Weird! But let’s install the agent manually:
[opc@compute-public1 ~]$ wget https://objectstorage.ca-toronto-1.oraclecloud.com/n/axmjwnk4dzjv/b/unified-monitoring-agent-centos7-repo/o/unified-monitoring-agent-0.0.5.rpm
--2020-10-07 15:29:16-- https://objectstorage.ca-toronto-1.oraclecloud.com/n/axmjwnk4dzjv/b/unified-monitoring-agent-centos7-repo/o/unified-monitoring-agent-0.0.5.rpm
Resolving objectstorage.ca-toronto-1.oraclecloud.com (objectstorage.ca-toronto-1.oraclecloud.com)... 134.70.72.3
Connecting to objectstorage.ca-toronto-1.oraclecloud.com (objectstorage.ca-toronto-1.oraclecloud.com)|134.70.72.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 36079287 (34M) [application/octet-stream]
Saving to: ‘unified-monitoring-agent-0.0.5.rpm’
100%[====================================================================================================================================================================================================>] 36,079,287 69.4MB/s in 0.5s
2020-10-07 15:29:17 (69.4 MB/s) - ‘unified-monitoring-agent-0.0.5.rpm’ saved [36079287/36079287]
[opc@compute-public1 ~]$ sudo rpm -ivh unified-monitoring-agent-0.0.5.rpm
Preparing... ################################# [100%]
Updating / installing...
1:unified-monitoring-agent-0.0.1+20################################# [100%]
Installing default conffile...
prelink detected. Installing /etc/prelink.conf.d/unified-monitoring-agent-ruby.conf ...
Installing logrotate.d config
Installing logrotate cron.d config
Configure unified-monitoring-agent to start, when booting up the OS...
Created symlink from /etc/systemd/system/multi-user.target.wants/unified-monitoring-agent.service to /usr/lib/systemd/system/unified-monitoring-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/unified-monitoring-agent_restarter.path to /usr/lib/systemd/system/unified-monitoring-agent_restarter.path.
Created symlink from /etc/systemd/system/multi-user.target.wants/unified-monitoring-agent_config_downloader.service to /usr/lib/systemd/system/unified-monitoring-agent_config_downloader.service.
Created symlink from /etc/systemd/system/timers.target.wants/unified-monitoring-agent_config_downloader.timer to /usr/lib/systemd/system/unified-monitoring-agent_config_downloader.timer.
Agent Install Complete.
[opc@compute-public1 ~]$ systemctl status unified-monitoring-agent.service
● unified-monitoring-agent.service - unified-monitoring-agent: Fluentd based data collector for Oracle Cloud Infrastructure SIEM
Loaded: loaded (/usr/lib/systemd/system/unified-monitoring-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-10-07 15:30:40 GMT; 39s ago
Docs: https://docs.cloud.oracle.com/
Process: 4388 ExecReload=/bin/kill -USR2 ${MAINPID} (code=exited, status=0/SUCCESS)
Process: 4256 ExecStart=/opt/unified-monitoring-agent/embedded/bin/fluentd --log /var/log/unified-monitoring-agent/unified-monitoring-agent.log --daemon /var/run/unified-monitoring-agent/unified-monitoring-agent.pid --log-rotate-size 1048576 --log-rotate-age 10 $EXTRA_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 4262 (fluentd)
Memory: 66.1M (limit: 5.0G)
CGroup: /system.slice/unified-monitoring-agent.service
├─4262 /opt/unified-monitoring-agent/embedded/bin/ruby /opt/unified-monitoring-agent/embedded/bin/fluentd --log /var/log/unified-monitoring-agent/unified-monitoring-agent.log --daemon /var/run/unified-monitoring-agent/unifie...
└─4265 /opt/unified-monitoring-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/unified-monitoring-agent/embedded/bin/fluentd --log /var/log/unified-monitoring-agent/unified-monitoring-agent.log --daemon /var/run/unified...
Oct 07 15:30:37 compute-public1 systemd[1]: Starting unified-monitoring-agent: Fluentd based data collector for Oracle Cloud Infrastructure SIEM...
Oct 07 15:30:40 compute-public1 systemd[1]: Started unified-monitoring-agent: Fluentd based data collector for Oracle Cloud Infrastructure SIEM.
Oct 07 15:30:54 compute-public1 systemd[1]: Reloading unified-monitoring-agent: Fluentd based data collector for Oracle Cloud Infrastructure SIEM.
Oct 07 15:30:54 compute-public1 systemd[1]: Reloaded unified-monitoring-agent: Fluentd based data collector for Oracle Cloud Infrastructure SIEM.
Oct 07 15:30:57 compute-public1 systemd[1]: Reloading unified-monitoring-agent: Fluentd based data collector for Oracle Cloud Infrastructure SIEM.
Oct 07 15:30:57 compute-public1 systemd[1]: Reloaded unified-monitoring-agent: Fluentd based data collector for Oracle Cloud Infrastructure SIEM.
Now the agent is there up and running! I observed same with OL8 image as well that the agent had to be installed manually.
One note on this, my instance is running in a public subnet so if you are doing this in private subnet you probably need to make sure you have Service Gateway configured.
Setting up logs
Next we will browse to Logging in OCI and start with Log Groups, for this I will fetch only one log to this group which will be /var/log/messages from the server.
Log Group only requires name and description as shown below.
After this I will create my Log, again I will give it a name only – multiple Logs can be assigned to a Log Group. Note the pre-requisites for creating Log.
I could also create entry for Service Logs (Object Storage, VCN) if I would need to.
Next is the Agent Configuration. I defined path which will be used to get the log and a distinct name for this. I’ve also defined Log which this Agent Configuration Belongs to.
Important! Remember you need to define a Dynamic Group which has the instances you want this Agent Configuration to use. You can easily include instances from specific compartment or using specific tag if you want to.
What I noticed next is that even though everything should be in order, I still had to restart the Linux side agent for it to work.
You can also view logs from Compute instance view by going to Custom Logs which is handy if someone doesn’t have access to Logging.
Summary
Setting up custom logs is fairly straightforward and seems this will be easy way to start utilizing your logs further in OCI!
Still there are minor things to be tweaked on, why aren’t my logs being synced to OCI? Which servers are supposed to be sending data and so on. It’s hard to debug if there is an issue and you need to go server by server to figure it out as OCI Console isn’t telling you much.
Also the Agent not being available in the OL image is something which might confuse in the start even though documentation says it should be there. Perhaps I missed some step there to get it installed!
From OCI documentation it states that it might take up to 25 minutes for the agent to be installed, this might be why you didn’t see the agent running
https://docs.cloud.oracle.com/en-us/iaas/Content/Logging/Reference/agent_troubleshooting.htm
Would you happen to know the difference between Management Agent and Unified Monitoring Agent?
I did review Oracle documentation but I’m still not able to figure out the real use case of having Management Agent
https://docs.oracle.com/en-us/iaas/management-agents/doc/install-management-agent-chapter.html
https://docs.oracle.com/en-us/iaas/Content/Logging/Concepts/agent_management.htm
I did end up reading another blog quite similar to this, wherein Management Agent are being used for OCI Logging
https://qiita.com/western24/items/547d21502bb985ae01f0