Hey All,
Having a bit of trouble with my network setup. I'm mostly a noob with very light understanding of what I'm working with, so bear with me.
I've got my opnsense box setup with mostly defaults set for rules. The opnsense box is hooked directly to an MB8611 modem on the WAN interface, with LAN interface running to an 8 port managed netgear switch on port 1 of the switch. I have port 2 of the switch hooked up to a vlan-aware access point. Other ports are occupied by physical links to some servers.
Everything on LAN works fine, including the access point. I have the access point setup with 3 SSIDs, all that can connect clients no problem. Some of the clients on the network are game consoles/gaming PCs that run into connectivity issues with some titles, I believe because of a strict NAT. Rather than just assigning outbound rules by static addresses, I opted to create a VLAN to house all gaming devices and segment them from the network. I don't need them to talk to each other or other devices.
I have created VLAN10, assigned it a gateway address of 192.168.10.0/24, setup DHCP and assigned the LAN as the parent interface. I created a new SSID on the access point and gave it the VLAN 10 tag. All of the ports on the switch are now assigned to default vlan1 with untagged traffic, and then ports 1 and 2 are assigned to vlan 10 with tagged traffic. Testing with an iPhone, this works totally fine. I get assigned the correct leases and can make outbound connections to the internet. Testing on a windows 11 gaming PC, previously connected to an untagged SSID and now switched over to the tagged SSID, I am unable to make outbound connections to the internet. Another android device that I connected with to the new SSID also worked fine, so I'm not sure what's up with this PC. I haven't tested other gaming devices yet.
Any ideas on what I'm doing wrong here?
Are you familiar with using Wireshark for traffic analysis? I think the next step is to figure out what is getting through and what isn't, to the Windows machine to start with.
Focusing on IPv4 for now, I would hope the network trace shows the DHCP request being sent out, the DHCP response with an IP for the Windows machine, and then some outbound web TCP traffic (eg google.com), followed by some sort of TCP response. But since it's not working, I imagine the latter would be replaced by -- ideally -- ICMP error messages that will describe the problem.
I'm familiar with wireshark, but don't have so much hands on experience with it. I'll give it a shot and see the type of responses I'm getting back from the afflicted machine.
In a the meantime, here's some of the firewall rules I have set on the interface itself as well as some floating rules. I'm following the recent guide from home network guy to set this up.
Looking at the firewall config, nothing stands out to me as unusual. On the gaming rules page, can you include the 16 autogenerated rules? I don't imagine that's where the issue is, but it might be worth a look.
When your Windows machine is attached on the VLAN network, you said it is successfully assigned an IPv4 address using DHCP, right? Is it able to ping the router? Can it ping anything successfully?
Really appreciate your help on this!
I've been messing with wireshark, but I'll admit I'm not super sure how to interpret it all. Biggest thing standing out is some TCP retransmission packets, but nothing jumping out as an immediate failure. I realized I'm having similar difficulties across devices I test on the vlan. I've been using my laptop, and I can ping things like google.com or just the DNS of 8.8.8.8 no problem. I can't ping the static router address of 192.168.10.1, but I think that's because of the rule I have in place that includes all private networks, which includes the vlan net. I also realized that on the interfaces overview section, I've got 1 collision error on the LAN, and 2 in/out errors on the vlan on the out side, but I'm not sure how to assess those. Also correct that I am getting the expected DHCP assignments on the vlan side.
Np, it helps me keep my networking skills fresh and relevant.
When you ping google.com, does this resolve as Google's v4 or V6 address? In either case, this at least proves that the VLAN routing is enough to: 1) reach the system's configured DNS server, 2) receive the DNS record, 3) send an ICMP (v6?) Echo to the default gateway, and 4) receive the ICMP Reply in response. If this works on v6, that makes sense since you have a rule explicitly for v6 ICMP to pass through. If this works on v4, I'm slightly confused why this works but nothing else does.
Which rule was this? But more importantly, in the Wireshark trace, does any traffic at all from 192.168.10.1 show up as a source IP? The pings from earlier, they only need the MAC address of the gateway. But the DHCP responses should be coming from 192.168.10.1. Does anything else come from that IP? On a related note, do you see any ARP broadcasts originating from your laptop asking for any addresses on the network, such as 192.168.10.1? I'm trying to rule out certain odd situations.
While collisions are unexpected in today's point-to-point switching topologies, if it's just in the single digits and the vast, vast number of total frames are passing through without issue, then this is not a cause for great concern about your L2 network. To be clear, are you running 1 Gbps on the OPNSense interface and on all the switch ports?
It's definitely returning the v4 address each ping.
The "only allow access to internet" rule on the gaming interface which encapsulates the firewall alias I set as "privatenetworks" that included the LAN and gaming nets. As far as wireshark, I do see traffic from 192.168.10.1 as a source! Being totally fresh with you on the ARP broadcasts, with my current understanding, I don't know if I'm picking it out right. I do see broadcast requests coming from my laptop to 192.168.10.1 via DNS with responses of AAAA ipv4only.arpa.
OPNSense has a 2.5 Gbps connection to from the modem to 2.5 Gbps port on the box itself. Then the switch that is connected to the LAN on both the OPNSense interface and the switch port are both 2.5 Gbps. The remainder of the ports on the switch are all also 2.5 Gbps capable, but there are some ports occupied by devices that only support a max of 1 Gbps.
I did test the vlan by disabling ipv6 entirely and bam! All traffic flows no problem. Certainly a quick fix, but for no reason other than looking to understand and learn, I do want to get it working. I've got both LAN and the vlan set to track interface, and originally, both to allow manual adjustment of DHCPv6 and router advertisements. That seems to work no problem on the LAN with a prefix ID of 1, passes all ipv6 tests. On the vlan, though, prefix ID of 2, I do get the expected ipv6 leases with the corresponding ID, but it can never pass the ipv6 tests.
It does appear that you have addressing working but not connectivity. As I said, I'm no expert on OPNSense but I did find this thread which has some thoughts: https://forum.opnsense.org/index.php?topic=29459.msg142330#msg142330
As well as:
This latter rule is... odd to me since there shouldn't really be NAT for IPv6 to a delegated prefix. But maybe that rule is meant to effectively disable the NAT and allow traffic to pass straight through without translation, obviously after applying your firewall rules.
Wow, that was a highly relevant thread! Feels like my search skills were lacking to not have come across that.
Seems like I'm only a couple of adjustments away from getting this working, so I'll give you some peace now. Thank you so much again for your time and advice!
Good luck! Also, when you do have everything working, back up your config. And also check to make sure your firewall is blocking inbound traffic as expected, for both v4 and v6.