Let me be honest - it is not so important for you to memorize this flowchart. It's not, it's not at all. But we're going to go through a flow here together. Just to describe all necessary steps that we follow depending on the type of circumstances or situation that might arise in our Open Shortest Path First, or OSPF, environment.
We have to have layer 3 connectivity to our adjacent devices for anything to work. Any routing protocol needs good layer 3. So I would be making sure that my interfaces are up/up, that they're addressed. And of course, included in this, as we have to agree about the subnet mask. OSPF will bark at you, if you disagree about the subnet mask being the common link between two adjacent routers. So show ip interface brief doesn't give us that. We might, in fact, want to look at that as well. So I would be looking at branch, I would also be looking at HQ.
Troubleshooting OSPF neighbor issues
Now why might it be necessary for us to ping, to make sure we have connectivity? Well here we're in a point-to-point scenario. So we'll see our interfaces up/up, as up/up will be connected. We'll ping to make sure that we have connectivity to that Internet Protocol, or IP, address. We had to be in the same subnet or else we won't form neighbor adjacencies.
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.2, timeout is 2 seconds:
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/9/10 ms
But what about a multi access environment? Are we going to be directly connected to that other router? Not necessarily. We've seen examples where we're directly connected over that Ethernet link. But what if we had a topology with four routers and we had switches in there providing the connectivity that we need? So we're no longer directly connected, we'd still be up/up. But do we officially have Layer 3 connectivity? So don't just assume you have Layer 3 connectivity. Execute a ping to make sure you have connectivity to those devices you will be forming a neighbor adjacency with.
Now this is not the most glorious command. But I'll tell you, it's one of the most examination useful commands, especially if you were to go into a troubleshooting oriented course.
Branch#show ip ospf interfaceSerial0/0/0 is up, line protocol is upInternet Address 192.168.1.1/30, Area 0, Attached via Network StatementProcess ID 1, Router ID 192.168.1.1, Network Type POINT_TO_POINT, Cost: 64...
show ip ospf interface and also show ip eigrp interface are both the best way to make sure that we're enabled for the interfaces that we expect. So we do this and if it comes up in the listing, that interface is enabled for OSPF. And so if we were challenged with forming a neighborship adjacency, we can do this and say, "Hey, am I locally configured?", or I'm not even running OSPF on that link because, maybe, my wildcard mask was something other than what I needed. My wildcard mask did not include that IP address that could've been a cause for this. We can also see the area, we'll cover that a little bit later. But of course the area has to match.
Troubleshooting neighbor adjacency involves making sure that both routers are configured properly. So we've just verified branch. Let's use the exact same command to verify HQ. So are my interfaces enabled for OSPF, same command, or up/up on Serial0/0/0. There is our IP address. And are we in the proper area? We are.
HQ#sh ip ospf interfaceSerial0/0/0 is up, line protocol is upInternet Address 192.168.1.2/30, Area 0Process ID 1, Router ID 192.168.1.2, Network Type POINT_TO_POINT, Cost: 64...
Here is a challenge for you. What if you couldn't go to HQ? What's a way that I could watch neighborship formation or potentially neighborship not being formed and observing that real-time? What would you be thinking about if you couldn't go to HQ or you're stuck on branch and really dealing with this? Because sometimes you don't have the luxury of telnetting or secure shelling into that other router in a timely way.
We could debug our OSPF packet and see if we are successfully sending "hello" packets and receiving "hello" packets. Yeah, and I might also do a debug IP OSPF adjacencies, where adjacencies is ADJ. So you could do a debug in the real world too. And that would be a pretty useful thing to do if you weren't able to do this command show ip ospf interface on HQ.
This probably isn't the best way to view your area if you've already done show ip ospf interface. Just look at the interface output, and it's got the area right in it. And actually, that's, in fact, going to be more telling because you could look at the show ip protocols output and you could be misled by the wildcard mask. Or if you've done the interface level command, it's going to come up as an interface here and could get confusing. So you know the lesson here is, we have to match the areas on all common OSPF routers in a broadcast domain. If we're running OSPF between five routers in a broadcast domain or here two, they better agree about the area.
Branch#show ip protocolsRouting Protocol is "ospf 1"...Routing for Networks:10.1.1.0 0.0.0.255 area 1192.168.1.0 0.0.0.3 area 0...
So in this case, branch area 0 and that was for interface serial 0/0/0. It's good, let's verify HQ now.
HQ#show ip protocolsRouting Protocol is "ospf 1"Routing for Networks:172.16.1.0 0.0.0.255 area 0192.168.1.0 0.0.0.3 area 0...
HQ's interface Serial0/0/0, also in area 0. All right, so we are still not neighbors and we've verified all these different parameters. What else can we verify?
Well are any of the interfaces passive? Passive interface serial 0/0/0 on either the branch router or HQ router? Where can we verify them if an interface is passive? show ip protocols will tell us. I don't see any information here on branch router indicating that serial 0/0/0 is passive.
Let's go check HQ then. Oops, problem identified, problem identified and you're going to quickly resolve this one. You got to remove that passive interface command.
HQ#sh ip protocolsRouting Protocol is "ospf 1"Routing for Networks:172.16.1.0 0.0.0.255 area 0192.168.1.0 0.0.0.3 area 0Passive Interface(s):Serial0/0/0...
So go into OSPF configuration mode, because that's where it's done. It's not in interface level command. No passive interface serial0/0/0, boom. Quickly, we've got to get one of those messages that says, "Loading to full."
*Jan 7 22:50:41.072: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.1.1 on Serial0/0/0 from LOADING to FULL, Loading Done
We're good, we've got our neighborship. So you look at this and you got to understand, there is one of those things that can break your adjacency just like we warned you before.
Troubleshoot OSPF routing table issues
So now we're neighbors. But we have another problem. We can't reach a specific network. Is HQ advertising that particular network? We can check this with show ip protocols - there is a row Routing for Networks.
HQ#show ip protocolsOutgoing update filter list for all interfaces is not setIncoming update filter list for all interfaces is not setRouter ID 192.168.1.2Number of areas in this router is 1. 1 normal 0 stub 0 nssaMaximum path: 4Routing for Networks:192.168.1.0 0.0.0.3 area 0
Do you see anything missing? I see that we're currently only routing for networks 192.168.1.0. So HQ wasn't configured properly to turn OSPF on that interface that is part of the 172.16.1.0/24 network.
This is kind of fun. Let's fix something that isn't broken. We can get there, we don't know why we can get there. That's kind of the problem. We are running OSPF. I can get to this network but I don't see it as an OSPF route. Well we could have probably figured that out by looking at the routing table. But if you want to get details about a route, you can do this command - show ip route and then you can put, in fact, the network. You can get some really powerful information. Things like routing tags show up here too. If it's like a redistributed route, you'd see tags and lots and lots of goodies. So you want to get hyper detailed information about a network as a routing entry, this is the place to go.
Branch#show ip route 172.16.1.0Routing entry for 172.16.1.0/24Known via "eigrp 1", distance 90, metric 2195456, type internalRedistributing via eigrp 1Last update from 192.168.1.2 on Serial1/0, 00:01:02 agoRouting Descriptor Blocks:* 192.168.1.2, from 192.168.1.2, 00:01:02 ago, via Serial0/0/0Route metric is 2195456, traffic share count is 1Total delay is 21000 microseconds, minimum bandwidth is 1544 KbitReliability 255/255, minimum MTU 1500 bytesLoading 1/255, Hops 1
So we are to set, "Oh! right, I've got EIGRP running on this network also." And EIGRP route will trump OSPF because it has the superior administrative distance of 90 versus 110. So this would be an interesting thing to resolve. I'm not going to tell you the right way to resolve this one because somehow we have EIGRP running also and we have to be very cautious at this point, because usually you don't want two different sources telling us the same information. But at least we've figured out why OSPF is not the reason why we can get there.
Troubleshoot OSPF path selection
In this troubleshooting example, we have traffic crossing both the Gigabit link as well the serial link at the same time.
How do we know this? Let's look at our routing table, show ip route ospf.
Branch#show ip route ospf172.16.0.0/24 is subnetted, 1 subnetsO 172.16.1.0 [110/2] via 18.104.22.168, 00:00:00, GigabitEthernet0/1[110/2] via 192.168.1.2, 00:00:00, Serial0/0/0
We can see that both paths are being utilized to get to 172.16.1.0 from branch. But wait, what is the default speed of a Gigabit Ethernet link? That's 1000, it's just 1000 megabits per second. What's the default for serial? It's 1.544 megabits per second. That has a cost of 64 by default. All right, so we have a cost of 1 by default and a cost of 64 by default. They're not the same, but we can see here in our routing table that the cost is listed as two to get to the destination network. So right there bells should be going off! Somebody's modified cost somewhere. So this proves the point - not all load-balancing is good, right? We have that pinhole congestion.
Branch#show ip ospf interfaceGigabitEthernet0/1 is up, line protocol is upInternet Address 22.214.171.124/27, Area 0, Attached via Network StatementProcess ID 1, Router ID 192.168.1.1, Network Type BROADCAST, Cost: 1...Serial0/0/0 is up, line protocol is upInternet Address 192.168.1.1/30, Area 0, Attached via Network StatementProcess ID 1, Router ID 192.168.1.1, Network Type POINT_TO_POINT, Cost: 1...
The gigabit link is basically slowed down effectively to the speed of the serial link and we just cause some massive quality problems for our traffic because we've probably got a bottleneck between the two places now and we're dropping just loads of traffic. That's probably what we would experience. Just massive bottlenecks, which are exhibited by dropped traffic because we get what is called "tail drop". Packets come in, router doesn't have room for it in memory buffers, because it's still waiting to send it down the gigabit link, it dies. So sometimes it's better to just go across one link.