What happens when a bunch of IETF super nerds show up in Paris for a major conference and discover their hotel's Wi-Fi network has imploded?
They give it an Extreme Wi-Fi Makeover.
The Internet Engineering Task Force (IETF), which sets a range of Internet standards, gathers for its 83rd meeting this week in Paris. The jam-packed agenda is filled with reports, presentations, and meetings of working groups, researchers, and birds-of-a-feather confabs. Among the topics: Multiparty Multimedia Session Control; Operational Security Capabilities for IP Network Infrastructure; and Worthwhile Extensible Internet Registration Data Service.
Despite it being springtime in Paris, such an agenda adds up to a lot of work for IETFers, all of whom also have their "day jobs" with a blue chip list of technology companies around the world. And that means that one thing that is even more important than a visit to Disneyland Paris (the No.2 most popular destination after Fontainebleau) is a hotel Wi-Fi network that works.
Flakey Wi-Fi, getting flakier
But as attendees began discovering on arrival last Sunday at the toney and towering Hotel Concorde Lafayette, close by the historic Champs-Elysees, the Wi-Fi was flakey and became flakier still as scores more attendees arrived and tried to connect.
Complaints began circulating quickly Sunday afternoon on the email list, although the spotty coverage, lost packets, and lack of a reliable or sometimes of any connection meant plenty of people didn't even see them. The wired network wasn't much better, apparently in part because in-room TVs shared the data connection.
"I've got what looks like a pretty good 802.11 connection, but am seeing about 30% packet loss. It's really not useable from my room as it is currently performing," noted attendee Ben Campbell.
"There are significant issues with this infrastructure, and it varies depending on where you are in the hotel and the load at the time," messaged Karen O'Donoghue, with the Internet Society. "I have seen similar performance numbers. For me it varied between 5% and 35% packet loss with latencies up to 5 or 6 seconds. That's obviously pretty painful."
One persistent complaint was that numerous outgoing Internet ports were inexplicably blocked, something which affected both wired and wireless connections. "The [Wi-Fi is] working well enough for me for skype, mail and browsing. One real issue (at least for me) is [that] outgoing non-standard TCP ports, which I need for my normal VPN, seem to be blocked," wrote Lou Berger, with LabN Consulting. "I am surprised by how many other ports the Concorde blocks," agreed Geoff Mulligan, chair of the IP for Small Objects (IPSO) Alliance.
One user complained that nearly all ports used by common instant messaging protocols are blocked, causing Cisco's Anton Ivanov to grouse, "This port blocking is so depressingly stupid.
I changed my iChat account settings to use port 443 for AIM and gmail.com Jabber, and now they're both working again."
Bailing wire and tape
Being engineers, users quickly got creative on their own to improve their wireless connectivity.
"There was no WiFi signal when on the desk in front of the window in my room, but after some experiments, I discovered that the signal was quite good... on the ceiling of the bathroom," emailed Marc Petit-Huguenin.
"I have a Nexus S phone, so I taped it on the ceiling of the bathroom, and used tethering over Bluetooth to bridge the gap to the desk," he explained. This is a slow connection, but good enough to send emails over SMTP or use vi [the popular Unix text editor] over SSH."
The hotel uses Wi-Fi gear from Colubris Networks, which since 2008 has been part of HP. The IETF attendees seemed in agreement that the problems were not caused by the Colubris equipment, but by its deployment and configuration, by an unnamed independent network integrator, who installed them. There were, they agreed, far too many access points, with radios set at high power, and poor channel planning. It was complicated, in the view of some attendees, by the hotel lacking the in-house IT resources to fix things.
But those resources did exist among the hotel's guests. Working behind the scenes, a team of IETF attendees negotiated with the hotel and were granted access to the wireless network by Sunday night. Chris Elliott, another Cisco veteran, reported to attendees by email late that night.
Reingineering on the fly
"[W]e've done a variety of configuration tests and changes," he wrote. "We believe we have improved the situation for most hotel guests. However, there may be instances where our changes (due to some holes in our understanding of where exactly some of the APs are located, for example) have made things worse in specific areas."
"This wireless network is still likely to have its issues," he warned. "I'm still seeing high latency and frequently dropped packets. It's improved, but far from perfect. It's a 2.4Ghz infrastructure in a highly 3D and rather radio transparent environment -- where the three non-overlapping channels [all that are possible in that band] are a real problem."
The changes made by the IETF makeover team included:
- Decreasing the AP receiver sensitivity ([changing] HP/Colubris configuration "distance" from "large" to "small");
- Increasing the minimum data and multicast rate from 1Mbps to 2Mbps;
- Decreasing the transmit power from 20dBm to 10dBm;
- And, turning off the radios on numerous APs to reduce the [RF] noise.
"In the process, we've hacked netdisco [a network management tool that maps MAC addresses to IP addresses to pinpoint switch ports] to be able to discover the hotel infrastructure and rancid [a free tool that monitors a device's configurations and maintains a history of changes in a Concurrent Version System (CVS) repository] to be able to at least minimally work with HP/Colubris APs, and added their private subnet to our management station to facilitate discovery, scripted changes, and monitoring," Elliott wrote, describing something close to a NOC trouble-shooting system put together on the fly.
A lot of attendees apparently saw immediate improvements.
"[M]y network connectivity all of a sudden got a whooooole lot better," emailed James Polk, also from Cisco. "I'd guess in the 12:30-1 a.m. timeframe. It was quite a surprise. Course, I expect a lot of folks are asleep (like I should be), so the contention should be less. That said, I was up at this time last night and the network was probably the worst yet."
"On a very quick check, it seems much better for my iPad and iPhone," wrote Ben Campbell. "Will check the laptop shortly."
Others reported no change.
"Happy to see that things appear fixed for some. It's not at all fixed for me on 5, though the problem could be me," reported Adam Montville, apparently with Tripwire, an IT security software company. "I am unable to join either wireless network (timeouts on both) [referring to the hotel Wi-Fi network proper and the IETF Wi-Fi network configured over it]. Not (yet) critical as I have a LAN line in the room and the appropriate adapter."
For some, the changes appear to have made things worse. "Connectivity was fine on the 32 floor for me on Sunday and on Monday morning," wrote Pat Thaler, of Broadcom. "After the message about the stuff they had done to make things better on the hotel networks, it has been very variable. Network strength goes from very good to very low or disconnected without moving my laptop. It's varying all over the scale. Finally pulled out my AP to use the wired connection so my VPN would stay up."
In a reply, Elliott reported the hotel was having power problems with network equipment of all types above the 27th floor.
Another variable was U.S.-made clients connecting to a French network, as Elliot alerted attendees in an email on Wednesday. "A quick note--some laptops will not associate to channels that are not allowed in the country they were built for," he said. "For example, US Apple Macs won't associate to channel 13. This is something that we've argued with Apple about--I believe it should be up to the AP to set the allowed channels and clients should be able to use them. I'm not worried about this in this case--folks should see other channels at acceptable signal strengths, and the Europeans, for example, will get a bit of a speed advantage."
Midweek "radical" changes
Elliot introduced additional changes early Wednesday morning.
"I've remapped the channels and frequency bands for all the APs on floors 2-33," he emailed his IETF colleagues. "This is a radical change, so please let me know how it works for you now."
Elliot noted that France lets Wi-Fi use channels 1-13 in the 2.4 GHz band. "As three channels are very limiting in a very 3D structure, like this hotel, I've chosen to go with 4 channels, using 1, 5, 9, and 13," he said. "This is a layout that is well respected by others, and one [that] we've considered using at the IETF on numerous occasions--and very similar to what we used in Hiroshima. You get a slight bit more of cross-channel interference, but the additional channel is worth it, especially in this hotel's environment."
Each floor now has approximately two access points on each of these four channels, with the channels staggered on adjacent floor. That design maximizes the distance between access points on the same channel. "I hope this will significantly improve the coverage in some rooms that had marginal or no signal while also improving the signal to noise ratio for all," he said
In addition, he switched a couple of the single-radio Colubris access points on each floor from 2.4 to 5 GHz, which would let at least laptops make use of one of four channels on the much less crowded band.
Stay tuned for further developments in this continuing saga.
John Cox covers wireless networking and mobile computing for Network World.
Twitter: http://twitter.com/johnwcoxnww
Blog RSS feed: http://www.networkworld.com/community/blog/2989/feed
Read more about anti-malware in Network World's Anti-malware section.