`netstat -p` on busybox

netstat is one of my favorite linux utilities and is always one of the first tools I use when starting to debug any network related issues. One of my favorite options that netstat provides is the -p flag. This flag is used to see which programs are talking on which sockets.

On to the real problem. I was investigating an issue with a OpenWrt router I was having which was that it was listening on a port which I wasn't expecting. I wanted to find the program which was holding this socket, but was disappointed to find that the netstat packaged with busybox does not have the -p option! Quelle Horreur!

To solve this, I decided to look into the proc filesystem to see if there was anything of value. The first thing I found was /proc/net/tcp which had some useful information but not the program holding it.

root@OpenWrt:~# cat /proc/net/udp sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode 34: 00000000:6922 00000000:0000 07 00000000:00000000 00:00000000 00000000 65534 0 2572701 2 80edd820 53: 00000000:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 1126 2 8099c820 67: 00000000:0043 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 1123 2 8099cc00

The port in question that I was investigating was 67 which is the DHCP port which I thought I had disabled, hence the investigation.

This file shows a number of things including the address/port combos for active sockets and metadata about those sockets. However, the most useful thing found in this file was that it lists the inode of the socket. Using the inode for the socket, I can find which process has a reference to it by walking the /proc/{pid}/ tree and looking for that inode in the /fd/ sub-tree for all running processes.

root@OpenWrt:~# find /proc/ -type l | grep /fd/ | xargs ls -la 2>/dev/null | grep 1123 lrwx------ 1 root root 64 Dec 12 19:28 /proc/707/fd/5 -> socket:[1123]

Ahah! Pid 707 is the culprit. Now just who is this pid?

root@OpenWrt:~# ps w PID USER VSZ STAT COMMAND 1 root 1432 S init 2 root 0 SW [keventd] 3 root 0 RWN [ksoftirqd_CPU0] 4 root 0 SW [kswapd] 5 root 0 SW [bdflush] 6 root 0 SW [kupdated] 8 root 0 SW [mtdblockd] 94 root 0 SWN [jffs2_gcd_mtd4] 118 root 1432 S init 154 root 1440 S syslogd -C16 156 root 1424 S klogd 676 root 1108 S /usr/sbin/dropbear -P /var/run/dropbear.1.pid -p 192. 684 root 976 S /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -t 707 nobody 884 S /usr/sbin/dnsmasq -K -D -y -Z -b -E -s lan -S /lan/ - 28001 root 1436 S udhcpc -t 0 -i eth0.1 -b -p /var/run/dhcp-eth0.1.pid 1573 root 1184 S /usr/sbin/dropbear -P /var/run/dropbear.1.pid -p 192. 1574 root 1440 S -ash 1873 root 1428 R ps w

Of course, it is dnsmasq. Now I have to find out why the configuration isn't taking hold. Anyways, it took longer to write this post than figure it out but I'm forcing myself to write more and I thought it was cool, so there it is.