so here comes the hot topic on internet for few days, that’s “Facebook Down”. Every other social media (company and users) have started trolling them.
Everyone knows that facebook was down on October 4th, 2021 on around 15:50 UTC. Everyone is talking that it was down so enough of that shit. Let’s talk talk about What, Why and How?
First of all, was facebook down actually?
The answer is YES. Yes it was down, not only facebook, but along with its wings instagram, messenger and whatsapp.
What actually Happened?
In short, Facebook’s server was not reachable.
In Details, after one sends access to facebook.com, DNS server responded with facebook’s IP and when trying to access to Facebook’s IP, it didn’t return anything.
The above picture shows how internet works when you send a simple request to access facebook from your browser. Here the problem was on Facebook’s DNS server.
At first, the browser request for ISP DNS (or whatever dns gateway you have set). Then the ISP DNS looks over the cache if it got it’s details, if not, it requests to root dns for the details. The root DNS server returns the respective TLD (.com/.net/.org etc) according to the TLD the domain got. Then again according to the root DNS server response, it request to the respective TLD’s DNS server, in this case its .com DNS server. It returns the ip according to the nameserver registered and for the third time, ISP DNS server requests to the nameserver and it finally get’s the ip of the respective server.
4 pair of request to find the server of related domain, and finally your browser can get the content from the server. This all happens in fraction of a second. and in this “Facebook Down” issue, it was that third pair which couldn’t be completed.
So, What was the issue?
Until here, we are sure that the issue was on network, not it’s time to understand the actual part that brought the error. According to a post from facebook engineering, They somehow messed up backbone routers that coordinate network traffic between facebook’s data centers and the users.
According to cloudflare, one of the largest DNS provider, referencing on blog from facebook, they mentioned that the issue was on the BGP (Border Gateway Protocol). BGP is the mechanism to exchange routing information between autonomous systems (AS) on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn’t know what to do, and the Internet wouldn’t work.
Comparing both the findings, we can conclude that Facebook Down was due to misconfiguration on their front facing server configuration.
This is a normal issue, anyone can face with simple mess on Gateway, server or BGP in this case. It might seem a simple issue, but once a configuration is messed up, it might be very difficult to catch up and make it available as it was before. If it was easy, a company like facebook with a good tech backup could have solved it easily and would not face this and “Facebook Down” wouldn’t be trending on twitter which won’t leave an opportunity like this to capitalize the market.
I assume you are a developer / system admin reading this blog. You might have faced this issue and might face this again in the future. Don’t panic, Even a large (I assume) group world class engineers working on a tech giant like facebook took around 6 hours to fix this. You working on some hulululu company for minimum wage might get a lot of burden on your company/startup, a lots of complains from customers, business, management, boss or people around. You need to keep calm and fix it. Stack overflow is a click away. just kick everyone out of your room and browse the internet for fixes.
To the people around that poor engineer, he/She didn’t do it intensionally. Your engineers are doing their best. They don’t want to miss their date, weekend or a sound sleep whatsoever fun thing waiting them out of those cubicles. You can help them by keeping calm and motivating.