Sunday, August 9, 2009

Firebox and new IPs

Trixie trixie bastards. So we gots some static ip's that are chillin behind our Firebox. The firebox itself has a dedicated ip.

0800 - Everything is working.
0830 - 1030 - Movin servers to new lo-cal.
1030 - 1100 - Reassign machine ips and tweak firebox.
1100 - 1930 - Not a god damn thing works.

So here is what we knew during this time:
- All machines could go out to the internets
- All servers were serving up their wares
- Internal pinging worked, outgoing pings worked
- 1 ip, behind the firebox on a drop, could be reached from external internets
- NONE of the simple firebox -> forward to machine from external sources worked
- All newly assigned ips worked, you could connect a server directly to the outside and bam
- Firebox logs showed incoming requests, denied, can't find destination
- All destinations were clearly mapped. No trickery. Come in on external ip xyz and forward to internal abc
- a trace route would die on the failing ips would die after the isp hop

So why? ARP CACHING!!!! Dirty, dirty, dirty girl. So what happened was this: We got assigned new ips. One went to the firebox and the rest behind it. The firebox was aggressive and broadcast saying "yo, here I be." The rest, chilling behind the firebox, did not. So, our isp was not able to properly route these requests.
The fix? Set every machine or ip on the open to the world to let it broadcast, then toss it back behind the firebox like a fat chick. ARP cache fixed.

This is why I'm not a network guy. This crap is lame. NullPointerExceptions are much easier. Or so I hear, since I've never had one.