Kurtis's Blog

Re-edited 01:08

I want to note one thing with the original text below - I don't think that .SE has done anything wrong, I am not even sure they would know how to notify the government portal (I know that I would know how to). Also, I don't think there is need for more monitoring. The fault was duly detected and quickly so. Monitoring with out understand the results will just lead to additonal misunderstandings. But I do think we need to think really hard about these events and how to communicate them in the future...

Tonight .SE had an error in the .SE zone file that rendered most .SE zones unreachable for almost an hour. Now I understand that generating and spreading information takes time - BUT this is what I have been arguing for a while. Today citizens look to the Internet to find out about and understand events.

23.15 CET the governments official web-site, all major news sources as well as the .SE registry themselves have yet to publish any info at all on what happened.

I believe that the right thing to do would have to publish a flash message, on the official web-siste www.krisinformation.se (that wasn't reachable through this time and I doubt many if any have it cached) - but so that once things start to work the information is again reachable.

The DNS system is very robust, and the distribution to the users is built so that all but one slave server can fail, but no system is 100% secure and against an error in the actual zone content it's hard to protect - except checks, checks again, and again checks. But, let's wait until we know more about the real reason for the outage before drawing conclusions on the root-cause. In the meantime, let's wait and see how long it takes for information to reach the public.

Oh, and what was the traffic effect? Around 10G...

UPDATE: I'll take one thing back - .SE had an announcement that they where doing maintenance between 19.00-23.00. Now we just lack the government....

23.43 and 23.46 first articles started to appear....still nothing on the offical gov site...or on the Swedish CERT site (which I wouldn't expect btw..)

I have travelled a lot and I know that you can have good and bad days Yesterday was a lot of the latter. Gold card checkin for SAS/Lufthansa at ARN took 40 minutes for two people, so by the time it was my turn I only got a middle seat at the second to last row.

This started worrying me as I only had an hour for connection in Frankfurt and then had to switch from A pier to B pier and then go through immigration and extended immigration. It turns out I didn't really have to worry about that...

We boarded in Stockholm at 6.20 (on-time) and then sat on the plane. Due to fog in Frankfurt we where issued a slot time, first for 8.15 and later at 9.00. We finally landed in Frankfurt at around 11.30. And of course one of the few planes that had left Frankfurt on time was the Beirut morning flight, my connection....

After queuing to change my ticket for an hour (And that was the short queue in the lounge) I realized my best option was to stay in Frankfurt and wait for the evening direct flight that leaves at 21.15...so I had a wonderful saturday in the Senator lounge...

When we finally boarded, of course the flight was overbooked, but I got on it. And to top the day off, two passengers didn't show up so we had to wait while they unloaded the luggage....

Anyway, at 1.30 I arrived in Beirut!

Somehow I have always been fascinated by Beirut, and I am really looking forward to the week here. I am here for MENOG5, the twice a year operational conference for the Middle East. MENOGs are always good content and discussions so I am really looking forward to the conference as well!

I am at MENOG5 where we are doing the pre-conference workshops. Together with Philip Smith I am teaching an IPv6 routing workshop working with students from the operators in the Middle East.

During the workshop yesterday, it Philip said something that made me realize how little time we have left to deploy IPv6.

The "original" IPv6 RFC1883 was published in December 1995. That is 14 years ago. If I look at the IPv4 Address Space report at www.potaroo.net we will see that

Projected IANA Unallocated Address Pool Exhaustion: 10-Nov-2011

Projected RIR Unallocated Address Pool Exhaustion: 22-Jan-2013

That means that we have 23 months left until IANA pool run-out. I yesterday twitted that we had as many months left as years we have been working on IPv6. Even if we count from the start of the IPng effort, that is not quite true. But 23 months is not a long time to get deployment going. The good news is what you see workshops such as the one that me and Philip are doing right now, is that it's actually not that hard to deploy. The cost is not that high as CAPEX is covered as part of normal upgrade cycles and backbone deployments can in many (most?) cases be done fairly quickly.

The problems are still with end-users / DSL deployments and lack of support, but that is coming as well.

First three days of MENOG5 was an IPv6 workshop, where we worked on real routers on deploying a dual-stack network with the students from all around the region.

Today will be somewhat more busy with two presentations. I just finished a tutorial on the IPv6 business case together with Philip Smith, and I will in a bit deliver the first presentation after the keynote. The second presentation will be about the history of peering and the success it brought in Europe.

On Thursday, several media sites where attacked with a DDoS attack, as well as the web-site of the Swedish police. The Swedish Civil Contingencies Agency (Myndigheten för Samhällsskydd och Beredskap) on Thursday announced that as both the police and the media where attacked at the same time, made the attacks more threatening to society and that they where monitoring the events. If it would have been needed they would have stepped in and co-ordinated the response from the authorities.

This statement scares me, shows how broken the planning in Sweden is and irritates me. All at the same time. Let me explain my views on this (if you haven't already heard them....)

Why was there a risk?

The first thing I react on in the comment above is that MSB clearly thinks that loosing access to media and the police at the same time would be critical. Why?

In reality the issue is that we have no, reliable, trusted and well-known communications channel over the Internet from the government to citizens. Yes, MSB runs the crisis information site, www.krisinformation.se, that is supposed to be this channel. The "only" problem with this site is that it's virtually unknown to any citizen. Unless that changes, and the government is prepared to truly do what it takes to make this known, the site is a waste of tax-payers money. I won't comment on wether I actually think the site works. The design is unknown to me, but given that it's not located at one of the major providers, I don't trust the capacity. Further, I am not an expert on web-building or ASP, but the code to me looks like it both have off-site dependencies as well as database calls.

So given the above, citizens today are forced to turn to media, a multitude of agencies etc to get information. Or forced to, it's the most natural thing to do....

What agency should deal with DDoS attacks

DDoS attacks are a crime. Crimes are investigated by the police, in normal order. This DDoS is no different. If MSB would have done their job properly and provided a well-known information channel, this should just have been any other matter for the police. The Swedish police also have very good resources and knowledgeable people, and are fully capable to deal with this (at least as far as I have seen in the past). In addition, there is the Swedish CERT, SITIC operated by the Post and Telecommnications regulator, that are able to provide assistance, knowledge and operational co-ordination between small operators and the larger ones (the larger ones have better direct co-ordination).

Why is this irritating?

MSB lacks operational knowledge and focus. In the quote above, they say their role is to co-ordinate between the authorities. As far as I know we talk about two authorities, the police and the CERT. Both of whom have excellent contacts. Why do we need a third agency for this? More, an agency that have clearly failed one of their most basic tasks.

This is what scares me. On Thursday, the government also gave MSB the task to come with a plan to protect Sweden against attacks over the Internet. Well, a first task would be to complete the work already given to them. What I don't understand is why this was not given to one of the agencies that are already operationally working on these issues. For example the regulator that have done an excellent job on contingency and resilience on telecom networks. My only guess is rivalry. MSB is part of the department of defence, the police part of the department of justice and the regulator part of the department of industry and trade. So the defence department have no role to play currently. At the same time, Internet attacks are becoming more important and I suspect more budgets are allocated to it. So you need to be part of it to get any money. This is what scares me. Instead of minimizing the people (and hence process and confusion involved) to a minimum and keeping the strategic decisions integrated into the operational roles, we are watching a game of rivalry. Sigh.