Selfhosted
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
view the rest of the comments
I would pay you money if you can fix this issue.
A number of people can't route to me at all. My domain is drkt.eu and sits on port 80 and 443 @ 89.150.135.135 and 2a05:f6c7:8039::1337
IPv6 is not big in my country, and I don't think anyone afflicted has IPv6 so I can't tell you if this affects both v4 and v6.
It's not a DNS issue, as the afflicted users can get the correct IPs from nslookup.
Mullvad VPN users are consistently unable to route to me.
One friend can't route to me from his workplace network. It's a small network and their admin claims they don't block anything so it's a mystery to him as well.
Another friend across town can't reach my network despite being so close to me hop-wise, but his network is run by wacks and is consequently also quite wack. I can't confidently say this is the same issue.
I'm not dropping any connections for any reason. My ISP claims they aren't doing any blocking of domains or IPs.
Traceroutes time out at consistent hops but it's different per afflicted network. The only recurring name has been costumer.tdc.net
It might not be related, but I can't route to catbox.moe and their admin says my IPs are not blacklisted in any of their systems.
What's your input?
You are correct; looking at the resolutions via Google's toolbox, it seems to be resolving fine.
I did a couple of traceroute tests from a couple locations (two data centres, my local WiFi, and via cellular data), and they all seem to end at
cpe.ae20-0.khk7nqp8.dk.customer.tdc.net. In teleco terms,cpeusually means "Client/Customer Premise Equipment", so it wouldn't surprise me if that is the address assigned to the network equipment closest to your server's local network (think neighborhood hub, or PON on premise, something super close to the demarcation point). If everyone else is able to get that far, then I'm more inclined to think the drop is happening on your equipment, not your ISP's equipment; but having said that, if this is a residential network (regardless if ISP is provisioning you gigabit connectivity via fibre or whatever), there will always be a likelihood for ISP to be doing more filtering.If you don't mind, what is the gateway you're using? Is there an ISP gateway and then your own, or straight from ISP equipment into your network? Are there any (single/double) NAT going on? Are there's any security/filtering options on (either) equipment(s)?
I have an ISP issued fiber modem that goes straight into my pfsense box. For IPv4 I have NAT and Firewall, for v6 it's just the firewall. It should be noted that I've been hosting with this setup for years and it wasn't a problem until I got this ISP at this location. I've had this ISP elsewhere without problems and another ISP here where it also wasn't a problem.
The traceroutes getting dropped was a misconfiguration in my firewall on my part, it should work now!
My friend across town can now ping me successfully but going to my website still gives him ERR_CONNECTION_TIMED_OUT in browser.
EDIT ----------
I got another troublesome user to run a quick test with the new firewall config. Also can't reach my website. Pings DO time out for him! Traceroute expectedly also fails, last hop was cpe.ae20-0.khk7nqp8.dk.costumer.tdc.net [87.61.121.169]
From my side, I now see 3
???s between the CPE and your IP address, which is also responding, so that's great.Can your friend do something like
curl -vvv https://drkt.euor whatever to see if the time out happens before/after SSL handshake etc.? Also, do they have any firewalls / security appliances configured to filter content? I'd be curious to seedigornslookupresult,pingortracerouteresult, andcurl -vvvresult, just to understand where it is breaking down.Also, do you have a login to your ISP's equipment? Are you able to set it to bridge mode to bypass it altogether? Just throwing ideas out there, to see if there is anything else on the go. That
cpedevice is also pretty curious for sure.Office friend still can't ping me. The campus friend can ping me but can't reach my website. I never trusted campus friend to be on a clean network so it's a bit fruitless to test his connection.
My ISP's equipment is entirely a modem right now, bridging straight to my pfsense router. It's not doing any weird filtering.
Here are the results of those from office friend
UPDATE
Office friends pinged me 4 times, all timed out on his end. His IP showed up in my state table and I captured these 2 packets
State table:
Packet capture: ~~(the only 2, despite the 4 pings, and they have bad checksums?)~~ All 8 expected packets show up but all do have bad checksums. I'm seeing a LOT of bad checksums. A known working connection from fourth friend did not show bad checksums in this same test.
He went to my domain while I was collecting packets from ANY protocol filtering his IP It's a bit long, so here's the cap file https://u.drkt.eu/iiUsH7.cap
I've learned that the bad checksums are to to be ignored because it's the NIC that's responsible for it, so tcpdump sees the wrong checksums and it doesn't matter. I have also learned that the Apache container actually does see the incoming connections. Here's an example of my working connection and his connection.
We're making progress!
Looks like connection are being made, so it isn't routing after all!
Looking at the first recording, based on the different packet length, I'd guess it is doing SSL handshake properly; whereas the second one seems to be all 0 length and so something is not working out. At least on a cursory glance, your settings seems to be pretty permissive, so unless your friend's using a super old system, it shouldn't be an issue. Do you know what OS your friend is using, and if it has up-to-date root certificates? Are they on a system with
opensslcli available? Judging by the unifi network, probably? Tryopenssl s_client -connect drkt.eu:443 -prexit(andctrl + cto quit after it stops) and see if you can see any oddities with the SSL handshake process.He's running Windows 10, unfortunately- but wouldn't SSL errors show up in Apache logs? His IP appears 0 times in all apache error and access logs dating back 8 months (the beginning of recorded logs).
Here's another example of a working request to https://drkt.eu, and his non-working request respectively.
See this page here that explains the
Flags: https://opensource.com/article/18/10/introduction-tcpdumpTypically, in a TCP connection, you'd SYN, SYN+ACK, ACK, then transfer actual data over. In the successful sequence, you see this happening as expected.
In the unsuccessful sequence, it seems to be stuck in SYN, SYN+ACK, but there is no ACK that follows (
Flags [.]).Where is the second one captured? On the user's system, or on your system? Something in between is determining the packet isn't intended for the destination and dropping it. It may be a firewall, it may be something else.
Thinking about this some more, could it be asymmetric routing?
If so, does this help?
Replying to both of your comments:
These are captured on both my gateway and the Apache LXC container. The captured packets are identical as far as tcpdump is aware on both of these systems. As far as I can tell, unless there are shenanigans at the firewall WAN NIC, this is how the packets arrive to my firewall.
And I don't think this is asymmetrical routing if I understand it correctly, as I only have one firewall. My interfaces are configured correctly according to that netgate article.
Sorry I got slammed by work last couple of days and didn’t check back.
I wonder if it could be asymmetrical routing by your ISP? You mentioned your setup was okay before but it doesn’t work since you changed location.
I think your friend with the UniFi network has a static IP. Can you try traceroute to their IP and see if the route is similar to the one taken by their ISP? I’m not sure if this is how you’d test for asymmetrical routing but if nothing else the symptoms sound similar.
This project has hit a bit of a dead end but I appreciate your input a lot. I may get an opportunity to run tcpdump from within their network soon- which is what I was waiting for and why I didn't reply yet, but things aren't really happening.
My ISP gave me an rDNS and I was off several of those dumb blocklists within the hour. One person who could not previously connect to me now can, so that was the issue for that user at least. They were using Mullvad VPN, so Mullvad blocks based on uceprotect or a similar blocklist.