Dan Farmer, security researcher and creator of the SATAN vulnerability scanner, teamed up with HD Moore, chief research officer at Rapid7 and lead architect of the Metasploit penetration testing framework found 230,000 publicly accessible Out-Of-Band management interfaces on the Internet. Many of these systems were running software that dates back to 2001.
Out-Of-Band server management
According to PCWorld, the Out-Of-Band (OOB) management interfaces expose servers to the Internet through microcontrollers embedded into the motherboard that run independently of the main OS and provide monitoring and administration functions. These microcontrollers are called Baseboard Management Controllers (BMCs). BMC’s are part of the Intelligent Platform Management Interface (IPMI), a standardized interface made up of a variety of sensors and controllers that allow administrators to manage servers remotely when they’re shut down or unresponsive, but are still connected to the power supply.
BMCs are embedded systems that have their own firmware—usually based on Linux. It’s an OS-agnostic and pervasive protocol. Initially developed by Intel (INTC), Dell (DELL), HP (HPQ), and other large equipment manufacturers. It was designed to help manage OOB or Lights-Out communication.
Rebranded by OEM manufacturers
Pure IPMI is usually implemented as a network service that runs on UDP port 623. It can either piggyback on the server’s network port or may use a dedicated Ethernet port. Vendors take IPMI as a base and add on a variety of services like mail, SNMP, and Web GUIs, and then rebrand the new package:
- Dell has iDRAC,
- Hewlett Packard iLO,
- IBM (IBM) IMM2
It’s also used as the engine for higher-level protocols. Some of the protocols are put out by the DMTF (WBEM, CIM, etc.) the OpenStack Foundation, and others. IPMI is particularly popular for large-scale provisioning, roll-outs, remote troubleshooting, and console access according to the research paper.
Parasitic oversight
The parasitic BMC has near-complete control and oversight of the server it rides upon. It can control the server’s including its memory, networking, and storage media. It can not be truly turned off. Instead, it runs continuously unless the power cord is completely pulled. An owner may only temporarily disable outside interaction unless you take a hammer to the motherboard.
Security researchers have warned in the past that most IPMI implementations suffer from architectural insecurities and other vulnerabilities/ These can be exploited to gain administrative access to BMCs. If attackers control the BMC they can mount attacks against the server’s OS as well as other servers from the same management group.
Dan Farmer stated in his recent paper Sold Down the River (PDF).
For over a decade major server manufacturers have harmed their customers by shipping servers that are vulnerable by default, with a management protocol that is insecure by design, and with little to no documentation about how to make things better … These vendors have not only gone out of their way to make their offerings difficult to understand or audit but also neglected to supply any substantial defense tools or helpful security controls.
Old BMC software
Mr. Farmer and Mr. Moore ran scans on the Internet in May 2014 and identified 230,000 publicly accessible BMCs. A deeper analysis of the at-risk systems revealed:
- 46.8% of them were running IPMI version 1.5, which dates back to 2001,
- 53.2% were running IPMI version 2.0, which was released in 2004.
The researchers reported that nearly all the systems running IPMI v1.5 were configured so that all accounts could be logged into without authentication. “… you can login to pretty much any older IPMI system without an account or a password.” Mr. Farmer explains this set-up can grant an attacker privileged access, “… in most cases, they grant administrative access, and even when they don’t the mere ability to execute any kind of commands without authentication is a bad thing.”
The team found that IPMI v.2.0, which includes cryptographic protection has its own security issues. For example, the first cipher option, known as cipher zero, provides no authentication, integrity, or confidentiality protection, Farmer said. A valid user name is required for logging in, without a password. The researcher found that around 60% of the publicly accessible BMCs running IPMI version 2 had this vulnerability.
Server management issues in IPMI 2.0
Another serious issue introduced by IPMI 2.0 stems from its RAKP key-exchange protocol that’s used when negotiating secure connections. The protocol allows an anonymous user to obtain password hashes associated with any accounts on the BMC, as long as the account names are known.
“This is an astonishingly bad design, because it allows an attacker to grab your password’s hash and do offline password cracking with as many resources as desired to throw at the problem,” Farmer said.
The analysis showed that 83% of the identified BMCs were vulnerable to this issue. A test with brute-force password guessing application John the Ripper, using a modest 4.7 million-word dictionary successfully cracked 30% of the BMC passwords. Farmer calculated that between 72.8 and 92.5% depending on password cracking success rate, of BMCs running IPMI 2.0 had authentication issues and were vulnerable to unauthorized access.
Canary in the coal mine
“While a quarter of a million BMCs is only a tiny sliver of the total computing power in the world, it’s still an important indicator as a kind of canary in the coal mine,” Mr. Farmer warns. He predicts that BMCs behind corporate firewalls share the same issues. He said. “While management systems are often not directly assailable from the outside they’re often left open once the outer thin hard candy shell of an organization is breached.”
The research paper includes recommendations for server administrators on how to mitigate some of the identified issues and better secure their BMCs. But the researcher concludes that ultimately the problem of insecure IPMI implementations will linger on for a long time. Mr. Farmer concludes with a rant:
Many of these problems would have been easy to fix if the IPMI protocol had undergone a serious security review or if the developers of modern BMCs had spent a little more effort in hardening their products and giving their customers the tools to secure their servers … At this point, it is far too late to effect meaningful change. The sheer number of servers that include a vulnerable BMC will guarantee that IPMI vulnerabilities and insecure configurations will continue to be a problem for years to come.
rb-
They told us so, about a year ago.
Defense-in-depth, block UDP port 623 at the perimeter – yes all of them, on the end-points, you are using personal firewalls?
Disable or remove the default vendor user names and pick a strong UID and PWD
Least privilege, the researchers warn that anyone who has administrative privileges on a BMC’s server has administrative control over it and may disable or enable IPMI, add or remove accounts, change the IP address, etc., etc.–all without any authentication to the BMC.
Related articles
Ralph Bach has been in IT long enough to know better and has blogged from his Bach Seat about IT, careers, and anything else that catches his attention since 2005. You can follow him on LinkedIn, Facebook, and Twitter. Email the Bach Seat here.
Proof that data breaches like Code Spaces, P.F.Chang’s, Domino’s, Target, Neiman Marcus continue to be inevitable. The Verge is reporting that the Heartbleed Open SSL bug is still running rampant. Despite the initial panic several months ago when Neel Mehta of Google’s (GOOG) security team discovered the major bug which put over a million web servers at risk, the threat is old news.
Being old news does not mean the problem’s solved according to the article. They cite security researcher Robert David Graham who found that at least 309,197 servers out there on the interwebs are still vulnerable to the exploit.
The Verge concludes that it’s likely that the lightly trod corners of the internet will remain vulnerable for many years to come, as sites with sub-par security standards continue to leave themselves and their users exposed. The danger is particularly real now since the exploit has been widely publicized. The bug, which affects the OpenSSL protocol used widely online, can cause some serious damage — it can be exploited to give hackers encryption keys, passwords, and other sensitive information.
In order to keep pace with the staff, CyBlock products can be set up to block Web access by categories and by half-hour so employees can access sports sites on their lunch break or after hours.
