I’m a huge fan of using tools available to help troubleshoot any issues there are. Luckily OCI has many free tools available such as Network Path Analyzer, which I’ve touched in the past.
This time we’re going to look on VCN Flow Logs, I think as a starting point we should be looking on enabling VCN Flow Logs to give you extra idea on what traffic is going around in your subnets.
Flow Logs actually helped me to troubleshoot one really weird issue with Oracle EBS using Load Balancer, one of the Oracle Forms page was calling additional EBS package, which in turn called one of the application server ports which we didn’t know. We hadn’t allowed traffic from Load Balancer subnet towards that port in application server subnet, and with help of Flow Logs, we could see connection getting rejected.
VCN Flow Logs can be enabled on different levels: VCN, subnet, VNIC or resources. In this example, I’m enabling them on two subnets.
When you go to subnet view on the OCI Console, you have option to browse into Logs. There’s a simple option to enable logs like below, one pre-requisite is that you have to have Log Group created (vcnflowlogs in below picture).
I’ve repeated same for my other subnet also.
I have two compute instances, one in each subnet. Let’s see what happens when I initiate SSH traffic from one compute VM to another and my subnet allows the traffic in the security list.
You can navigate by clicking the Log Name and it will take you to OCI Logging Service to respective log as below.
I expect traffic now to be allowed from source to destination and looking the logs below explore log, I see a line appearing few minutes after.
I want to repeat same test and remove the security list rule allowing traffic to port 22 on destination server, sure enough, SSH connection times out and I can see below in Flow Logs.
On the right side you have down arrow which let’s you drill down into details of the event, when I expand the row I see all data that is available.
Packet details in VCN Flow Logs
What Oracle has done and what has helped me troubleshooting, is also filtering the data – just right click on specific row and you get option to filter for example by port 22 or you could use to filter all data where action is REJECT.
Further on you can also group data for example by IP, Action etc. Useful to see if you have one problematic server for example, whatever you are trying to solve!
Summary
Enabling VCN Flow Logs for troubleshooting and monitoring should be one of the actions you take on building the infrastructure. Above simple examples are meant to show you how easy it is to do, and how quickly you can get logs enabled to help you. You can always ingest logs to some third party system as well when needed.
==> Case:
In my tenancy (“trymanish”):
() I have a “default” domain with a user (“trymanish@outlook.com”) created as administrator , that I use this to login to the OCI console.
() Now, I also created another domain “Non-Prod-Dev”, with a compartment “Non-Prod-Dev-Compartment1”
() I have 2 users created there in domain “Non-Prod-Dev”:
— One user (“mani.aggarwal”) created as default administrator.
— Second user “ocpmanish” created in group “OSadmin”.
() Now, I login into newly created domain “Non-Prod-Dev” with user (“mani.aggarwal”) created as default administrator.
() Then, I try to create a policy as “OSpolicy” and assign it to group “OSadmin” for all users in that group (including the user created above as “ocpmanish”).
— Policy command as below: (this is to allow bucket management to the users in the group)
“Allow group ‘Non-Prod-Dev’/’OSadmin’ to manage object-family in compartment Non-Prod-Dev-Compartment1”
— I get an error and it does not create the policy.
() But, when I try login to “default” domain in my tenancy “trymanish” with user (“trymanish@outlook.com”) created as administrator.
() I am able to create the policy mentioned above.
==> Conclusion: Is my undersstanding correct ?
() In order to create policy, do I need to login as admin user in the default domain of my tenancy and then create it under either root or any other compartment I want to ?
I think you need to give more permissions for the non-default domain administrator. See: Granting users or groups the identity domain administrator role for domains other than the default domain grants them full administrator permissions to only that domain (not to the tenancy). At least one administrator for the identity domain must be granted the identity domain administrator role directly.
Thanks, I was able to fix it.
Below is the problem statement and solution, just FYI:-
==>Problem:
() In an “OCI cloud login window 1”, a tenancy “trymanish”, has an existing domain “Default”, with an admin user “trymanish@outlook.com”.
() Now, I created a new domain “Non-Prod-Dev”, a new compartment “Non-Prod-Dev-Compartment1”, and a default admin user “mani.aggarwal”.
() The user “mani.aggarwal” by default gets assigned to the group “Domain_Administrators”.
() Now, I open another “OCI cloud login window 2”, and I login to the tenancy “trymanish”, domain “Non-Prod-Dev” using the new user “mani.aggarwal”.
() As I login, and go to policy, I am unable to see any policy or even create a new policy in the new compartment “Non-Prod-Dev-Compartment1”.
() Shows error as below:
() But, when I go back to “OCI cloud login window 1”, where I am logged into the same tenancy “trymanish”, with the default domain “Default”
()– and admin user “trymanish@outlook.com”, guess what, I am able to create the policy for the group “Domain_Administrators”,
()– in the new domain “Non-Prod-Dev”, with new compartment “Non-Prod-Dev-Compartment1”, and the default admin user “mani.aggarwal”.
==>Solution:
() Login to tenancy “trymanish” from the default admin user “trymanish@outlook.com”.
() Go to the new compartment that was created as “Non-Prod-Dev-Compartment1”, then select the newly created domain “Non-Prod-Dev”.
() Then create a new group “Administrators” and add the new admin user “mani.aggarwal” to the group.
() Select the new domain “Non-Prod-Dev” and then “Policies” –> “Create Policy”.
() Policy name “resource-admin”, Description “resource admin for domain Non-Prod-Dev”,
() In principal, the created group “Administrators” will cascade the policy to its users.
() Now logout the user “mani.aggarwal” that was logged in from the new domain “Non-Prod-Dev” and relogin.
() Click on policies and now you will be able to see policies and also should be able to create new policy also.