Troubleshooting Network Problems in OCI using VCN Flow logs

In our earlier article, we discussed how to enable VCN Flow Logs in OCI. Once the logs are enabled, the next step is to use them for troubleshooting.

This is where VCN Flow Logs become very useful.

When a connection does not work, we look for did the traffic reach the target? was it allowed? was it rejected? Which source IP and port were involved? Which destination IP and port were involved?

VCN Flow Logs help us answer these questions because they record details about traffic that has been accepted or rejected based on the security rules in the VCN. These logs are useful for auditing traffic and for troubleshooting security lists and NSGs. Each flow log record contains traffic information for a single VNIC.

Why VCN Flow Logs Help in Troubleshooting

Mostly when network connection fails, the issue can be caused by:

·        a missing ingress rule

·        a missing egress rule

·        a wrong port

·        traffic going to the wrong IP

·        an NSG or security list blocking the traffic

VCN Flow Logs help us look at actual traffic metadata. In simple terms, VCN Flow Logs give us a traffic story.

Open the Log and Use “Explore with Log Search”

Once flow logs are enabled and traffic has started flowing, we can open the log details page in OCI Logging. Click Observability & Management -> Logging -> Search.

Also, we can invoke the search from the Logs page. Choose the Log (which has been enabled for VCN/Subnet/Resource as discussed in the Enablement point in the previous article) and then click Explore log and Explore with Log search as shown in the picture.

This opens the Search page with that specific log already selected, so we can perform deeper analysis and investigation directly on the Search page.

This is one of the most useful options for troubleshooting because it takes us from a simple log view into a search and analysis view.

When we open Explore with Log Search, OCI Logging lets us do several useful things.

the Logging Search page can:

·        search logs in basic or advanced mode

·        filter by fields, text, log groups, and time range

·        visualize results

·        open each log line in raw JSON

·        export search results to JSON

 

The Logging Search page supports time-based filtering, and searches can be limited to a selected time range. OCI also notes that log searches are available only for a 14-day range.

We have VCN Flow logs enabled on a Subnet. Let’s try to connect an instance created in that subnet and then check the logs whether it has been captured and what all other options or information we can collect from logs.

We have connected to the instance. Now we could see Flow log has started capturing the events.

Let’s try Explore log search also.

The Logging Search supports both Basic mode and Advanced mode. Basic mode lets us use filters directly in the interface, while Advanced mode lets us type custom queries. The default advanced query is:

search "ocid1.tenancy.oc1..<unique_id>" | sort by datetime desc

The Log search page has list of events happened in the particular time frame. By default, it will list events seen in last 5 minutes. We can increase and also choose custom timeframe.  If we expand any event, it will list the details in JSON format.

Important Fields to Check in VCN Flow Logs

There are several fields in a VCN Flow Log record. It will be shown in JSON format. In JSON view, we can expand fields, inspect values, and copy the data.

Some of the most useful ones for troubleshooting are these:

data.action

This tells us whether the traffic was ACCEPT or REJECT. Oracle says ACCEPT means the traffic was accepted by the security lists, and REJECT means it was rejected.

data.sourceAddress

This shows the IP address of the source. This helps us confirm where the traffic came from and which host started the traffic. The source IP address is in IPv4 or IPv6 notation.

data.sourcePort

This shows the source port used by the traffic. This helps us see which source port was used.

data.destinationAddress

This shows the target IP address. The destination IP address is in IPv4 or IPv6 notation.

data.destinationPort

This shows the destination port. This is one of the most useful troubleshooting fields. It tells us which service port the traffic was trying to reach, such as:

·        22 for SSH

·        80 for HTTP

·        443 for HTTPS

·        1521 for Oracle database

data.protocolName

This shows the protocol name such as TCP. It also includes the protocol number in data.protocol.

data.packets and data.bytesOut

These fields help us understand how much traffic was recorded in the capture window.

data.status

This tells us whether the log record is normal or if data was missing. Oracle documents OK, NODATA, and SKIPDATA as possible values.

Lets take an example

·        data.action = REJECT

·        data.sourceAddress = "45.33.12.122"

·        data.destinationAddress = "10.0.0.41"

·        data.destinationPort = 111

·        data.protocolName = TCP

it usually means traffic from 45.33.12.122 to 10.0.0.41 on port 111 was blocked by network rules. It makes clear that the action field shows whether the traffic was accepted or rejected based on VCN security rules.

In the Explore Log search we can filter the search using the attributes for example data.sourceaddress or data.sourceport.

In the above screenshot, we have done the log search for specific destination ipaddress. This helps us quickly narrow the log data without typing complicated queries.

We can use filter the search events by clicking left click on the list.

Filter matching – it will list all events matching to the chosen event in the defined timeframe.

Filter not matching – it will list all events omitting the chosen event in the defined timeframe.

Example use cases: SSH connection is failing

If SSH to a private instance is not working, we can search for records where the destination port is 22. Then we can check:

• source IP
• destination IP
• action = ACCEPT or REJECT

If the action is REJECT, that points us toward NSG or security list rules.

Application port is blocked

If an application should listen on port 8080 or 443, we can filter for that destination port and see whether the traffic was accepted or rejected. The destination port and action fields are especially useful here.