Do you have any more information?

Yes. Ref: #11973

@jingyih Should "endpoint health" not simply call '/health' endpoint on each member instead of performing a quorum read on cluster?

i can take look at this issue but the problem is am not that familiar with the project. so can anyone guide me through this

@jingyih Should "endpoint health" not simply call '/health' endpoint on each member instead of performing a quorum read on cluster?

I think by design etcdctl uses clientv3 library, for example the KV API [1]. The library communicates with etcd server via gRPC, not via HTTP. I think we just need to add additional call to make sure there is no alarm raised at the server by calling the clinetv3 maintenance API [2].

[1]

[2]

@gyuho @spzala Just want to double check with you. Is the current behavior of etcdctl endpoint health by design? I think if the server has alarms, this should return false.

$ etcdctl put foo bar
{"level":"warn","ts":"2020-06-22T02:44:40.368-0700","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-1fb6632a-4ac3-4f99-a1c3-c84860cddfb1/127.0.0.1:2379","attempt":0,"error":"rpc error: code = ResourceExhausted desc = etcdserver: mvcc: database space exceeded"}
Error: etcdserver: mvcc: database space exceeded

$ etcdctl endpoint health -w json
[{"endpoint":"127.0.0.1:2379","health":true,"took":"1.339424ms"}]

use endpoint status can do it .

$ ./bin/etcdctl endpoint status -w json  
[{"Endpoint":"127.0.0.1:2379","Status":{"header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":1,"raft_term":4},"version":"3.5.0-pre","dbSize":24576,"leader":10276657743932975437,"raftIndex":11,"raftTerm":4,"raftAppliedIndex":11,"errors":["memberID:10276657743932975437 alarm:NOSPACE "],"dbSizeInUse":24576}}]

and /health is a metric api,not a standard api.

@jingyih Should "endpoint health" not simply call '/health' endpoint on each member instead of performing a quorum read on cluster?

I think by design etcdctl uses clientv3 library, for example the KV API [1]. The library communicates with etcd server via gRPC, not via HTTP. I think we just need to add additional call to make sure there is no alarm raised at the server by calling the clinetv3 maintenance API [2].

[1]

etcd/clientv3/kv.go

Line 33 in ebdcced

type KV interface {

[2]

etcd/clientv3/maintenance.go

Lines 36 to 38 in ebdcced

	type Maintenance interface {
	// AlarmList gets all active alarms.
	AlarmList(ctx context.Context) (*AlarmResponse, error)

@jingyih I have made the change and issued a pull request. /cc @gyuho @xiang90