What steps will reproduce the problem? 1. ./filestore-upload-test -weed=:9333 -parallel.write=5 -parallel.read=2 -request.size.init=500000 2>&1 | grep ER

What is the expected output? What do you see instead? Instead of empty (no error) I see: 2012/11/11 21:47:05 ERROR: error uploading: error POSTing to http://localhost:8080/3,01b47ff6a24911: errorcode=500 message={"error":"Failed to write to replicas for volume 3","size":0}

What version of the product are you using? On what operating system? 0.23 (01f2fcf72daac3e13cd9a664aaae125f4202e603)

Please provide any additional information below. master: gthomas@waterhouse:~/projects/go/src/github.com/tgulacsi/filestore-upload-test$ mkdir -p /tmp/weed ; ./weed volume -dir=/tmp/weed 2012/11/11 21:43:48 Volume Folder permission: -rwxr-xr-x 2012/11/11 21:43:48 Loading index file 1.idx size 64432 2012/11/11 21:43:48 In dir /tmp/weed reads volume = 1 , replicationType = 000 2012/11/11 21:43:48 Loading index file 2.idx size 62592 2012/11/11 21:43:48 In dir /tmp/weed reads volume = 2 , replicationType = 000 2012/11/11 21:43:48 Loading index file 3.idx size 64384 2012/11/11 21:43:48 In dir /tmp/weed reads volume = 3 , replicationType = 000 2012/11/11 21:43:48 Loading index file 4.idx size 63376 2012/11/11 21:43:48 In dir /tmp/weed reads volume = 4 , replicationType = 000 2012/11/11 21:43:48 Loading index file 5.idx size 63824 2012/11/11 21:43:48 In dir /tmp/weed reads volume = 5 , replicationType = 000 2012/11/11 21:43:48 Store started on dir: /tmp/weed with 5 volumes 2012/11/11 21:43:48 store joined at localhost:9333 2012/11/11 21:43:48 Start Weed volume server 0.23 at http://localhost:8080 2012/11/11 21:47:05 post to http://localhost:9333/dir/lookup EOF 2012/11/11 21:47:05 Failed to lookup for 3 EOF

Seems the unnecessary master lookup for replication for 1-copy data caused this issue.

Fixed in the git head. Going to release a new version soon.

with ecd0399f8d3e60b6f065258b92901e5b06faffe9:

tgulacsi@nell:~/projects/weed-fs$ mkdir -p /tmp/weed ; ./weed master -mdir=/tmp/weed & ./weed volume -dir=/tmp/weed [1] 15212 2012/11/12 10:51:05 Volume Folder permission: -rwxr-xr-x 2012/11/12 10:51:05 In dir /tmp/weed reads volume = 1 , replicationType = 000 2012/11/12 10:51:05 In dir /tmp/weed reads volume = 2 , replicationType = 000 2012/11/12 10:51:05 In dir /tmp/weed reads volume = 3 , replicationType = 000 2012/11/12 10:51:05 In dir /tmp/weed reads volume = 4 , replicationType = 000 2012/11/12 10:51:05 In dir /tmp/weed reads volume = 5 , replicationType = 000 2012/11/12 10:51:05 Store started on dir: /tmp/weed with 5 volumes 2012/11/12 10:51:05 store joined at localhost:9333 2012/11/12 10:51:05 Start Weed volume server 0.24 at http://localhost:8080 2012/11/12 10:51:05 Loading file id sequence 20000 => 30000 2012/11/12 10:51:05 Volume Size Limit is 32768 MB 2012/11/12 10:51:05 Start Weed Master 0.24 at port 9333 topo adds child DefaultDataCenter topo:DefaultDataCenter adds child DefaultRack topo:DefaultDataCenter:DefaultRack adds child localhost:8080 2012/11/12 10:51:28 Saving file id sequence 40000 to /tmp/weed/weed.seq

Another window:

./filestore-upload-test -weed=:9333 -parallel.write=5 -parallel.read=2 -request.size.init=500000 2>&1 2012/11/12 10:52:49 starting round 1... 2012/11/12 10:52:49 calling uploadRound 2012/11/12 10:52:49 POST http://localhost:8080/5,7533b405235e response: {"error":"Failed to write to local disk","size":500000} 2012/11/12 10:52:49 CU err=error POSTing to http://localhost:8080/5,7533b405235e: errorcode=500 message={"error":"Failed to write to local disk","size":500000} 2012/11/12 10:52:49 uploadRound: error uploading: error POSTing to http://localhost:8080/5,7533b405235e: errorcode=500 message={"error":"Failed to write to local disk","size":500000} 2012/11/12 10:52:49 error with round 1: error uploading: error POSTing to http://localhost:8080/5,7533b405235e: errorcode=500 message={"error":"Failed to write to local disk","size":500000}

:(

Thanks. It is displaying the wrong error message. Now it is corrected in git head.

gthomas@waterhouse:~/projects/go/src/github.com/tgulacsi/filestore-upload-test$ ./filestore-upload-test -request.size.init=512000 -request.num=1000 -parallel.read=2 -parallel.write=5 -weed=:9333 2>&1|grep ER 2012/11/12 21:45:27 ERROR: error uploading: error POSTing to http://localhost:8080/2,2c9dee3ecb3f: error POSTing &{Method:POST URL:http://localhost:8080/2,2c9dee3ecb3f Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[ident] Content-Type:[multipart/form-data; boundary=b362c614208f8bae73efe7bae72472089dcdce160ccb2a15ca85dbbc7cf4] Mime-Version:[1.0]] Body:{Reader:0xf8414f0f00} ContentLength:512243 TransferEncoding:[] Close:false Host:localhost:8080 Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr: RequestURI: TLS:}: lookup localhost: no such host

what is this lookup localhost ?

Comment deleted

I can not pin-point this error. I got different errors when trying to reproduce, but they all seem before hitting the weedfs volume server.

For your particular error, the error is to lookup "localhost" before visiting "http://localhost:8080/2,2c9dee3ecb3f". Inside WeedFS, there are no http calls(for this setup). So this "localhost" lookup can only come from the client itself.

I just cannot make this new version work:

tgulacsi@nell:~/projects/weed-fs$ rm -rf /tmp/weed ; mkdir -p /tmp/weed/master ; ./we master -mdir=/tmp/weed/master -debug=true & ./weed volume -dir=/tmp/weed -max=1 -debug=true [1] 27149 2012/11/13 15:41:31 Volume Folder permission: -rwxr-xr-x 2012/11/13 15:41:31 Store started on dir: /tmp/weed with 0 volumes 2012/11/13 15:41:31 store joined at localhost:9333 2012/11/13 15:41:31 Start Weed volume server 0.24 at http://localhost:8080 2012/11/13 15:41:31 post to http://localhost:9333/dir/join dial tcp 127.0.0.1:9333: connection refused 2012/11/13 15:41:31 Setting file id sequence 10000 2012/11/13 15:41:31 Volume Size Limit is 32768 MB 2012/11/13 15:41:31 Start Weed Master 0.24 at port 9333 topo adds child DefaultDataCenter topo:DefaultDataCenter adds child DefaultRack topo:DefaultDataCenter:DefaultRack adds child localhost:8080

2012/11/13 15:41:45 In dir /tmp/weed adds volume = 1 , replicationType = 000 [assign volume = 1 , replicationType = 000 , error = ] Created Volume 1 on topo:DefaultDataCenter:DefaultRack:localhost:8080 2012/11/13 15:41:45 Saving file id sequence 20000 to /tmp/weed/master/weed.seq [assign volume = 1 , replicationType = 000 , error = Volume Id 1 already exists!] Failed to assign 1 to [topo:DefaultDataCenter:DefaultRack:localhost:8080] [assign volume = 1 , replicationType = 000 , error = Volume Id 1 already exists!] Failed to assign 1 to [topo:DefaultDataCenter:DefaultRack:localhost:8080] [assign volume = 1 , replicationType = 000 , error = Volume Id 1 already exists!] Failed to assign 1 to [topo:DefaultDataCenter:DefaultRack:localhost:8080] [assign volume = 1 , replicationType = 000 , error = Volume Id 1 already exists!] Failed to assign 1 to [topo:DefaultDataCenter:DefaultRack:localhost:8080]

By the way: how do you compile weed-fs? I needed to modify every import of "pkg/ to "../../pkg/

GThomas

Hi, GThomas, I can not reproduce your issue. But I added some more locking in order to avoid similar issue.

To compile: 1) set the GOPATH to the root of weed-fs 2) go to $GOPATH/src/cmd/weed/, type in "go build -o ../../../bin/weed"

Thanks, that worked.

Still see some upload error when concurrency is greater than 2 - I'll have to investigate a little more - how do I enable copious logging?

Another question, which I couldn't answer by reading the Beaver paper: do Haystack/Weed-FS store the uploaded file's mime type? I think its essential to be able to serve the file (usually not photo :)) back to browsers unaltered. Would this need to be added to the needle structure? Would you accept such modification?

When starting volume server, there is an option readTimeout. The default value is 1 second, which may be too low for the test.

I tried setting it to 3 seconds, and got no error now.

With readTimeout=3, I have no errors. Thank you!