In order to see what is going happening on the network, I captured some tcpdumps. Reading the files in Wireshark, does not show any Gluster specifics. It seems that Wireshark does not know how to decode (or rather dissect) the Gluster traffic. Very unfortunate, as quite some future troubleshooting and performance analysis may require investigating the network packets.
Luckily the Wireshark Developer's guide contains a chapter on Adding a basic dissector. After writing some code and tests, I now have some Wireshark packages that recognize some Gluster communication. The RPMs are available for testing, feedback over email is appreciated.
With the updated packages, the output of tshark (the terminal version of Wireshark) identifies some Gluster packets:
$ tshark -r gluster-communication.cap 'tcp.len > 0' | head
7 0.002572 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
8 0.002633 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
11 0.002909 192.168.122.137 -> 192.168.122.184 Gluster Dump V1 DUMP Reply (Call In 7)
12 0.002918 192.168.122.137 -> 192.168.122.184 Gluster Dump V1 DUMP Reply (Call In 8)
15 0.003104 192.168.122.184 -> 192.168.122.137 Gluster Portmap V1 PORTBYBRICK Call
16 0.003158 192.168.122.184 -> 192.168.122.137 Gluster Portmap V1 PORTBYBRICK Call
17 0.003298 192.168.122.137 -> 192.168.122.184 Gluster Portmap V1 PORTBYBRICK Reply (Call In 15)
18 0.003310 192.168.122.137 -> 192.168.122.184 Gluster Portmap V1 PORTBYBRICK Reply (Call In 16)
31 3.013909 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
32 3.013965 192.168.122.184 -> 192.168.122.137 PCEP Unknown Message (0). As with several other protocols, Wireshark detects some packets as non-gluster ones. In this tcpdump, there surely is no PCEP traffic (last line in the above output). Each dissector for a protocol should do some sanity checks to see if a packet belongs to its protocol. These checks are not easy to do, and hence quite some protocols detect packets from Gluster as their communication stream.
Luckily it is possible to disable a protocol in the
~/.wireshark/disabled_protos file. Finding the correct names of a protocol isn't always straight forward. Use Wireshark to graphically create the file is the easiest, it also takes care of disabling the protocols that are possibly encapsulated. In Wireshark- go to Analyze in the menu
- click "Enabled Protools"
- uncheck PCEP (and while you are at it, also uncheck SSL as it gives the same issues)
After these steps,
tshark should recognize all traffic to and from port 24007 as belonging to one of the Gluster protocols. I have only tested the Wireshark dissectors on Gluster 3.2.5, later releases use some newer versions of some protocols and these may not be detected yet.