I have five hosts to form weed-fs cluster.

At first I've tried using "server" command to start both master-server and volume-server. After encountering problems I've switched to separate master, volume processes.

Problem manifests itself as following.

When starting three or more volume servers, I've noticed that "dir/status" call lists only two of them. When I restart any volume server it doesn't join master. Number of volume servers in the list remains the same. What happens is that publicIp field and a name of one of servers changes, but ip field remains the same.

It seems to me that weed-fs doesn't correctly handle proxying. Master server considers two volume servers with the same ip-address as duplicates. When follower-master forwards request from volume server to leader-master, leader-master sees follower-master as a request originator and uses it's ip-address instead of volume server's ip-address. As a result of wrong ip-address assignment some volume servers are unable to join master, they are considered as duplicated.

My guess is that http.RemoteAddr field in go standard library doesn't support X-Forwarded-For header by default. Support for X-Forwarded-For should be rolled manually to fix this issue (see http://osdir.com/ml/go-language-discuss/2012-10/msg02165.html and https://groups.google.com/forum/#!msg/golang-nuts/lomWKs0kOfE/gwmoD3cTxesJ)

Thanks for the details! I would need to see your network topology diagram to understand this case. The dir/status would also be helpful.

In topology/configuration.go file, line 49, the Locate(ip, datacenter, rack) function assumes IP address is unique. But it seems your case has multiple volume servers on the same IP address?

No each volume server has distinct IP-address.

The problem is that IP-address is erroneously detected.

It goes like this:

When master-servers are started on all three servers everything is ok: all master-servers join the same cluster and leader is elected:

When I start volume server

• it connects not to leader master-server but to random master-server from cluster.
• chosen master-server forwards join-request from volume server up to leader master-server.
• leader master server confuses ip-address of volume server. It get ip-address of join-request originator as volume server ip-address, but it's not. It's an ip-address of some randomly chosen master server from cluster.
• when two volume servers chooses same master server they pushes each other from cluster because leader master server considers that they have same ip-address.

I see. How did you start the volume server? Did you also specify -ip="..." ?

• indy

  • weed -v 10 -log_dir=/opt/weedfs/log/volume volume -dir=./storage -port=8083 -mserver=indy:9333 -publicIp=indy

  • weed -v 10 -log_dir=/opt/weedfs/log/master master -mdir=./storage -port=9333 -publicIp=indy -defaultReplication=001 -peers=austin:9333,indy:9333,marina:9333,nasca:9333,valencia:9333

• valencia

  • weed -v 10 -log_dir=/opt/weedfs/log/volume volume -dir=./storage -port=8083 -mserver=valencia:9333 -publicIp=valencia

  • weed -v 10 -log_dir=/opt/weedfs/log/master master -mdir=./storage -port=9333 -publicIp=valencia -defaultReplication=001 -peers=austin:9333,indy:9333,marina:9333,nasca:9333,valencia:9333

• nasca

  • weed -v 10 -log_dir=/opt/weedfs/log/volume volume -dir=./storage -port=8083 -mserver=nasca:9333 -publicIp=nasca

  • weed -v 10 -log_dir=/opt/weedfs/log/master master -mdir=./storage -port=9333 -publicIp=nasca -defaultReplication=001 -peers=austin:9333,indy:9333,marina:9333,nasca:9333,valencia:9333

I've just tried --ip option and it works fine with explicit ip.

When --ip option is specified for volume server it listens to connections on this specific ip. It would be more convinient if it listens on 0.0.0.0 ip for my use case.

My use case is a case of network misadministration. Hosts has several ip-addresses, hosts are referenced by names from /etc/hosts file and there is a possibility that single host is referenced with two different ip addresses (/etc/hosts files are out of sync).

It may be better to have three options:

--listen-on-ip (ip address for socket to listen on) --internal-ip (ip address to pass out in weed-fs messages to master) --external-ip (public ip address to use when downloading files through web, current --publicIp)

I'd like to notice that there is still a bug. Weed-FS doesn't work without --ip flag in distributed master setup. Remote address detection should be fixed by adding support for X-Forwarded-For or --ip flag should be made mandatory.

Added "-ip.bind" option to volume servers.