Cross Connect Across OCI Tenancies through VCN Peering

In OCI, workloads are often distributed across different tenancies for reasons such as organizational separation or project isolation. In such cases, there may still be a requirement for private communication between resources hosted in separate tenancies. We will see how we can make the cross connectivity across regions.

In this exercise the tenancies are in the same region. We have used DRG to make the connectivity between them. Since both VCNs are in the same OCI region, this exercise uses a cross-tenancy VCN attachment to a DRG. RPC is used for DRG-to-DRG remote peering scenarios, commonly across regions or where separate DRGs are peered. In this article, we are not using RPC.

 First create IAM policy and define policy statements in both tenancies. Basically, we are making the tenancies know each other.

For this exercise we name the tenancies as below.

Requestor tenancy: owns the DRG.
Acceptor tenancy: owns the VCN that will be attached to the requestor DRG.

As per the prerequisites, we need OCID of both the tenancies and we need an IAM group on both the side to manage this attachment. In this exercise we have chosen Administrators group as we are going to have this setting at tenancy level.  DRG and VCN are available already in the respective tenancies. For connectivity testing purpose, we can windows VM at requestor tenancy and a Database VM in the acceptor tenancy.

Source Tenancy Name : Requestor
Requestor OCID:  ocid1.tenancy.oc1..aaaaaaaacq5gprsqz26em4koaokcntrpey4adi7mzzgve53cm44ozbxa4z4a
IAM Group in Requestor: ocid1.group.oc1..aaaaaaaa5xhfg6q6pefhbvwmtvzelih2kiuxgjwf5bg6bfqseeewjapd5mha

VCN CIDR Range: 192.168.0.0/16

Target Tenancy Name:   Acceptor
Acceptor OCID: ocid1.tenancy.oc1..aaaaaaaargwveli3zwyyeywzeex6x4zxqodjkbvnde4la7j5cckwioxqfncq
IAM Group name in Acceptor:  ocid1.group.oc1..aaaaaaaat4idcm3dbat2zlgqar3wzvxkz4hqlxynwkyo3gztsbtgg4on2xhq

VCN CIDR Range: 10.0.0.0/16

 

At the requestor tenancy, Goto OCI Console -> Identity & Security -> Policy.

Create Policy in root compartment. But if there is a requirement that peering has to be done between particular compartment, then we can create the policy in those compartments.

Click Policy editor and provide below policy statements.

Define tenancy AcceptorTenancy as ocid1.tenancy.oc1..aaaaaaaargwveli3zwyyeywzeex6x4zxqodjkbvnde4la7j5cckwioxqfncq

Define group VCN-Admin as ocid1.group.oc1..aaaaaaaat4idcm3dbat2zlgqar3wzvxkz4hqlxynwkyo3gztsbtgg4on2xhq

Endorse group Default/Administrators to manage drg-attachment in tenancy AcceptorTenancy

Admit group VCN-Admin of tenancy AcceptorTenancy to manage drg in tenancy

Endorse group Default/Administrators to manage virtual-network-family in tenancy AcceptorTenancy

 

Let me explain the policy statements.

First statement defines the Acceptor tenancy OCID.
Second statement defines VCN-Admin as OCID of group available at acceptor
Third statement endorses that at acceptor tenancy the requestor tenancy group (administrators) can manage DRG attachment. This is required since the VCN at the acceptor going to be attached with DRG of requestor as shown in the picture.
Fourth statement is equivalent of third, but in the reverse order.
Fifth statement is required to make the group to manage virtual network family in acceptor tenancy. I was getting Privilege error without this statement. I suggest not to ignore this policy.

 

After policy are entered, click create button to create the policy.

At the Acceptor tenancy, Goto OCI Console -> Identity & Security -> Policy.

Create Policy in root compartment. But if there is a requirement that peering has to be done between particular compartment, then we can create the policy in those compartments.

Click Policy editor and provide below policy statements.

Define tenancy RequestorTenancy as ocid1.tenancy.oc1..aaaaaaaacq5gprsqz26em4koaokcntrpey4adi7mzzgve53cm44ozbxa4z4a

Define group DRG-Admin as ocid1.group.oc1..aaaaaaaa5xhfg6q6pefhbvwmtvzelih2kiuxgjwf5bg6bfqseeewjapd5mha

Admit group DRG-Admin of tenancy RequestorTenancy to manage virtual-network-family in tenancy

Admit group DRG-Admin of tenancy RequestorTenancy to manage drg-attachment in tenancy

Endorse group Default/Administrators to manage drg in tenancy RequestorTenancy

Let me explain the policy statements.

First statement defines the Requestor tenancy OCID.
Second statement defines DRG-Admin as OCID of group available at requestor
Third statement admits that DRG-Admin of tenancy can manage network in acceptor tenancy. Again, I suggest not to this policy. I was getting privilege error without this policy.

Fourth statement admits that DRG-Admin can also manage drg-attachment in acceptor tenancy.

Fifth statement is endorses acceptor group (administrators) can manage DRG in requestor tenancy.

 

Click Create and complete the policy creation.

Now we have created policies at both tenancies and we can go ahead and create the connectivity.

In the Acceptor tenancy, open the DBVCN VCN.
Go to Resources → Dynamic Routing Gateway Attachments.

Click Create DRG Attachment.

Choose Another tenancy.
Provide the DRG OCID from the Requestor tenancy.
Create the attachment.

It will create Attachment with DRG.

You can find the cross-tenancy column as “Yes”.

The tenancies are connected now. You can find the same in DRG of requestor.

In the requestor tenancy, Go to Networking -> Customer Connectivity -> Dynamic Routing Gateway.

Choose the DRG. In the DRG page, move to Attachments section. We could see VCN attachments.

This confirms that cross-tenancy connection is successful at requestor side as well. But currently only connection has established, but Security list and Route table needs to be updated to allow the connection s between these tenancies. For quick lab validation, we temporarily allowed broader traffic. For production, restrict the rules to the required ports and source CIDRs.

Security list – Requestor

Route table – Requestor

Destination CIDR: 10.0.0.0/16
Target: DRG

Security list – Acceptor

Route table – Acceptor

Destination CIDR: 192.168.0.0/16
Target: DRG

Connectivity testing

Requestor Windows VM to VM database connectivity.

 

VM Database server to Windows VM

 

This completes the cross-tenancy network foundation. In the next article, we will build on this setup and create a database link from an OCI Autonomous Database private endpoint to the VM-based Oracle Database across tenancies.

Troubleshooting Network Problems in OCI using VCN Flow logs

In our earlier article, we discussed how to enable VCN Flow Logs in OCI. Once the logs are enabled, the next step is to use them for troubleshooting.

This is where VCN Flow Logs become very useful.

When a connection does not work, we look for did the traffic reach the target? was it allowed? was it rejected? Which source IP and port were involved? Which destination IP and port were involved?

VCN Flow Logs help us answer these questions because they record details about traffic that has been accepted or rejected based on the security rules in the VCN. These logs are useful for auditing traffic and for troubleshooting security lists and NSGs. Each flow log record contains traffic information for a single VNIC.

Why VCN Flow Logs Help in Troubleshooting

Mostly when network connection fails, the issue can be caused by:

·        a missing ingress rule

·        a missing egress rule

·        a wrong port

·        traffic going to the wrong IP

·        an NSG or security list blocking the traffic

VCN Flow Logs help us look at actual traffic metadata. In simple terms, VCN Flow Logs give us a traffic story.

Open the Log and Use “Explore with Log Search”

Once flow logs are enabled and traffic has started flowing, we can open the log details page in OCI Logging. Click Observability & Management -> Logging -> Search.

Also, we can invoke the search from the Logs page. Choose the Log (which has been enabled for VCN/Subnet/Resource as discussed in the Enablement point in the previous article) and then click Explore log and Explore with Log search as shown in the picture.

This opens the Search page with that specific log already selected, so we can perform deeper analysis and investigation directly on the Search page.

This is one of the most useful options for troubleshooting because it takes us from a simple log view into a search and analysis view.

When we open Explore with Log Search, OCI Logging lets us do several useful things.

the Logging Search page can:

·        search logs in basic or advanced mode

·        filter by fields, text, log groups, and time range

·        visualize results

·        open each log line in raw JSON

·        export search results to JSON

 

The Logging Search page supports time-based filtering, and searches can be limited to a selected time range. OCI also notes that log searches are available only for a 14-day range.

We have VCN Flow logs enabled on a Subnet. Let’s try to connect an instance created in that subnet and then check the logs whether it has been captured and what all other options or information we can collect from logs.

We have connected to the instance. Now we could see Flow log has started capturing the events.

Let’s try Explore log search also.

The Logging Search supports both Basic mode and Advanced mode. Basic mode lets us use filters directly in the interface, while Advanced mode lets us type custom queries. The default advanced query is:

search "ocid1.tenancy.oc1..<unique_id>" | sort by datetime desc

The Log search page has list of events happened in the particular time frame. By default, it will list events seen in last 5 minutes. We can increase and also choose custom timeframe.  If we expand any event, it will list the details in JSON format.

Important Fields to Check in VCN Flow Logs

There are several fields in a VCN Flow Log record. It will be shown in JSON format. In JSON view, we can expand fields, inspect values, and copy the data.

Some of the most useful ones for troubleshooting are these:

data.action

This tells us whether the traffic was ACCEPT or REJECT. Oracle says ACCEPT means the traffic was accepted by the security lists, and REJECT means it was rejected.

data.sourceAddress

This shows the IP address of the source. This helps us confirm where the traffic came from and which host started the traffic. The source IP address is in IPv4 or IPv6 notation.

data.sourcePort

This shows the source port used by the traffic. This helps us see which source port was used.

data.destinationAddress

This shows the target IP address. The destination IP address is in IPv4 or IPv6 notation.

data.destinationPort

This shows the destination port. This is one of the most useful troubleshooting fields. It tells us which service port the traffic was trying to reach, such as:

·        22 for SSH

·        80 for HTTP

·        443 for HTTPS

·        1521 for Oracle database

data.protocolName

This shows the protocol name such as TCP. It also includes the protocol number in data.protocol.

data.packets and data.bytesOut

These fields help us understand how much traffic was recorded in the capture window.

data.status

This tells us whether the log record is normal or if data was missing. Oracle documents OK, NODATA, and SKIPDATA as possible values.

Lets take an example

·        data.action = REJECT

·        data.sourceAddress = "45.33.12.122"

·        data.destinationAddress = "10.0.0.41"

·        data.destinationPort = 111

·        data.protocolName = TCP

it usually means traffic from 45.33.12.122 to 10.0.0.41 on port 111 was blocked by network rules. It makes clear that the action field shows whether the traffic was accepted or rejected based on VCN security rules.

In the Explore Log search we can filter the search using the attributes for example data.sourceaddress or data.sourceport.

In the above screenshot, we have done the log search for specific destination ipaddress. This helps us quickly narrow the log data without typing complicated queries.

We can use filter the search events by clicking left click on the list.

Filter matching – it will list all events matching to the chosen event in the defined timeframe.

Filter not matching – it will list all events omitting the chosen event in the defined timeframe.

Example use cases: SSH connection is failing

If SSH to a private instance is not working, we can search for records where the destination port is 22. Then we can check:

• source IP
• destination IP
• action = ACCEPT or REJECT

If the action is REJECT, that points us toward NSG or security list rules.

Application port is blocked

If an application should listen on port 8080 or 443, we can filter for that destination port and see whether the traffic was accepted or rejected. The destination port and action fields are especially useful here.

 VCN Flow Logs

In OCI, communication between resources depends on the Virtual Cloud Network (VCN) configuration. Because of this, enabling network logging is important for monitoring traffic, debugging connectivity problems, and troubleshooting network issues.

In this article, we discuss the prerequisites and the enablement of VCN Flow Logs. In the next article, we will see how VCN Flow Logs can be used to troubleshoot network problems. VCN Flow Logs show network traffic details inside OCI and help us understand whether traffic was allowed or rejected. VCN Flow Logs are useful for auditing traffic and for troubleshooting security lists and NSGs.

When a connection does not work, we usually look for did information related to traffic status, source IP and Destination IP got involved in the communication. VCN Flow Logs help us answer these questions. Oracle says they record details about traffic that passes through a VCN and can help troubleshoot security lists and NSGs.

We could also consider flow logs as a Network traffic diary.

Flow logs are stored in the Logging service, so we first need a log group.

Create a Log Group

Logically a log group is like a folder that keeps related logs together.  Log group can be invoked from Observability & Management -> Logging -> Log groups.

We can create a log group by clicking “Create log group” button.

After that, the log group will be visible in OCI.

 

The next step is Create a capture fitler. Capture filter is a networking component that defines rules to selectively include or exclude specific traffic for VCN flow logs. It helps control costs and improve security monitoring by filtering traffic based on protocols, IP addresses, or ports before it is logged or mirrored.

Capture filter can be invoked from Networking -> Network command center -> Capture filters.

The capture filters support:

·        a sampling rate - Helps to control how much network traffic is captured in the flow logs.

·        include rules - Allow to capture only the traffic that matches specific conditions.

·        exclude rules - Helps ignore traffic that we do not want to capture in the flow logs.

·        filters based on CIDR, protocol, and traffic direction - Helps to narrow the captured traffic by source or destination network, type of protocol, and whether the traffic is incoming or outgoing.

 

 

Enable the Flow Log

In the OCI Console, we open Networking and then go to Network Command Center. From there, we open Flow logs.

In the Flow logs page, Click Enable Flow logs button.

Now we review the settings and enable the flow log.

The Flow creation requires a name, destination which is Log group and Filter to customize the logging and then the Enablement Points which is source for the Flow log.

The Enablement point cloud be VCN or Subnet or particular resource like targeting instances, load balancers, or network load balancers.  

VCN – It will enable flow logs to all VNIC registered with the VCN.

Subnet – It enables flow logs to VNIC which are registered with specific subnet.

Resources – We could specifically choose particular Instance, load balancer or Network load balancer.

For our exercise, lets go with subnet option. First let us choose a subnet which is not enabled with flow logging.

Now add enablement points as subnet and Enable the flow logs.

We could see the flow logs has been created for chosen subnet.

The same has been reflected in the subnet page (Subnet deails -> Monitoring)

At this stage the Subnet has been enabled with logging.  We will discuss about Troubleshooting the subnet traffic in the next article.