Skip to content
This repository was archived by the owner on Mar 26, 2020. It is now read-only.

Commit 49f2c8d

Browse files
rishubhjainrishubhjain
committed
Merge branch 'master' of https://github.com/gluster/glusterd2 into cluster-wide-options
2 parents f1d991b + 5f8ec37 commit 49f2c8d

23 files changed

Lines changed: 307 additions & 40 deletions

File tree

‎doc/profiling.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
Tracking resource usage in Glusterd2
2+
====================================
3+
4+
Tracing and profiling with `pprof` is intended for developers only. This can be
5+
used for debugging memory leaks/consumption, slow flows through the code and so
6+
on. Users will normally not want profiling enabled on their production systems.
7+
To investigate memory allocations in Glusterd2, it is needed to enable the
8+
profiling feature. This can be done by adding `"profiling": true` in
9+
the `--config` file.
10+
Enabling profiling makes standard Golang pprof endpoints available. For memory
11+
allocations `/debug/pprof/heap` is most useful.
12+
Capturing a snapshot of the current allocations in the Glusterd2 is pretty
13+
simple. On the node running Glusterd2, the go pprof tool command can be used:
14+
```
15+
[root@fedora3 glusterd2]# go tool pprof http://localhost:24007/debug/pprof/heap
16+
File: glusterd2
17+
Build ID: 7a94c2e498445577aaf7f910d6ef1c3adc19d553
18+
Type: inuse_space
19+
Time: Nov 28, 2018 at 2:55pm (IST)
20+
Entering interactive mode (type "help" for commands, "o" for options)
21+
(pprof)
22+
```
23+
24+
For working with the size of the allocations, it is helpful to set all sizes to
25+
`megabytes`, otherwise `auto` is used as a `unit` and that would add human
26+
readable B, kB, MB postfixes. This however is not useful for sorting with
27+
scripts. So, set the `unit` to `megabytes` instead:
28+
29+
```
30+
(pprof) unit=megabytes
31+
```
32+
33+
```
34+
(pprof) top
35+
Showing nodes accounting for 330.75MB, 98.52% of 335.73MB total
36+
Dropped 305 nodes (cum <= 1.68MB)
37+
Showing top 10 nodes out of 14
38+
flat flat% sum% cum cum%
39+
326MB 97.10% 97.10% 326MB 97.10% github.com/gluster/glusterd2/pkg/sunrpc.ReadFullRecord /root/work/src/github.com/gluster/glusterd2/pkg/sunrpc/record.go
40+
2.38MB 0.71% 97.81% 2.38MB 0.71% github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/rafthttp.startStreamWriter /root/work/src/github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/rafthttp/stream.go
41+
2.38MB 0.71% 98.52% 4.77MB 1.42% github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/rafthttp.startPeer /root/work/src/github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/rafthttp/peer.go
42+
0 0% 98.52% 326.01MB 97.10% github.com/gluster/glusterd2/pkg/sunrpc.(*serverCodec).ReadRequestHeader /root/work/src/github.com/gluster/glusterd2/pkg/sunrpc/servercodec.go
43+
0 0% 98.52% 4.83MB 1.44% github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).apply /root/work/src/github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/etcdserver/server.go
44+
0 0% 98.52% 4.83MB 1.44% github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyAll /root/work/src/github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/etcdserver/server.go
45+
0 0% 98.52% 4.77MB 1.42% github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyConfChange /root/work/src/github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/etcdserver/server.go
46+
0 0% 98.52% 4.83MB 1.44% github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntries /root/work/src/github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/etcdserver/server.go
47+
0 0% 98.52% 4.83MB 1.44% github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).run.func6 /root/work/src/github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/etcdserver/server.go
48+
0 0% 98.52% 4.83MB 1.44% github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/pkg/schedule.(*fifo).run /root/work/src/github.com/gluster/glusterd2/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go
49+
(pprof)
50+
```
51+
52+
Looking at the above output, we can see that the first line consumes hundreds
53+
of MB while the other lines are just a fraction of it.
54+
This makes sure that ReadFullRecord is consuming more memory than required.
55+
To understand things even more clearly, we can see which line of ReadFullRecord
56+
consumes the memory. To view that use the list command:
57+
58+
```
59+
(pprof) list ReadFullRecord
60+
Total: 335.73MB
61+
ROUTINE ======================== github.com/gluster/glusterd2/pkg/sunrpc.ReadFullRecord in /root/work/src/github.com/gluster/glusterd2/pkg/sunrpc/clientcodec.go
62+
8.11kB 8.11kB (flat, cum) 0.0024% of Total
63+
. . 1:package sunrpc
64+
. . 2:
65+
. . 3:import (
66+
8.11kB 8.11kB 4: "bytes"
67+
. . 5: "io"
68+
. . 6: "net"
69+
. . 7: "net/rpc"
70+
. . 8: "sync"
71+
. . 9:
72+
ROUTINE ======================== github.com/gluster/glusterd2/pkg/sunrpc.ReadFullRecord in /root/work/src/github.com/gluster/glusterd2/pkg/sunrpc/record.go
73+
326MB 326MB (flat, cum) 97.10% of Total
74+
. . 96:func ReadFullRecord(conn io.Reader) ([]byte, error) {
75+
. . 97:
76+
. . 98: // In almost all cases, RPC message contain only one fragment which
77+
. . 99: // is not too big in size. But set a cap on buffer size to prevent
78+
. . 100: // rogue clients from filling up memory.
79+
326MB 326MB 101: record := bytes.NewBuffer(make([]byte, 0, maxRecordSize))
80+
. . 102: var fragmentHeader uint32
81+
. . 103: for {
82+
. . 104: // Read record fragment header
83+
. . 105: err := binary.Read(conn, binary.BigEndian, &fragmentHeader)
84+
. . 106: if err != nil {
85+
(pprof)
86+
```
87+
88+
The records allocation is the culprit and we need to look further into
89+
the code as to why the records is allocating so much and fix it.

‎e2e/smartvol_ops_test.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -498,6 +498,7 @@ func editDevice(t *testing.T) {
498498
var peerID string
499499
for _, peer := range peerList {
500500
deviceList, err = client.DeviceList(peer.ID.String(), "")
501+
r.Nil(err)
501502
if len(deviceList) > 0 {
502503
peerID = peer.ID.String()
503504
break

‎glustercli/cmd/cluster-options.go‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
package cmd
2+
3+
import (
4+
"errors"
5+
"fmt"
6+
"os"
7+
8+
"github.com/gluster/glusterd2/pkg/api"
9+
10+
"github.com/olekukonko/tablewriter"
11+
log "github.com/sirupsen/logrus"
12+
"github.com/spf13/cobra"
13+
)
14+
15+
const (
16+
helpClusterOptionCmd = "Gluster Cluster Option Management"
17+
helpClusterOptionSetCmd = "Set Cluster Option"
18+
helpClusterOptionGetCmd = "Get Cluster Option"
19+
)
20+
21+
func init() {
22+
clusterCmd.AddCommand(clusterOptionGetCmd)
23+
clusterCmd.AddCommand(clusterOptionSetCmd)
24+
}
25+
26+
var clusterCmd = &cobra.Command{
27+
Use: "cluster",
28+
Short: helpClusterOptionCmd,
29+
}
30+
31+
var clusterOptionSetCmd = &cobra.Command{
32+
Use: "set <option> <value> [<option> <value>]...",
33+
Short: helpClusterOptionSetCmd,
34+
Args: clusterSetCmdArgs,
35+
Run: clusterSetCmdRun,
36+
}
37+
38+
func clusterSetCmdArgs(cmd *cobra.Command, args []string) error {
39+
// Ensure we have enough arguments for the command
40+
if len(args) < 2 {
41+
return errors.New("need at least 2 arguments")
42+
}
43+
44+
// Ensure we have a proper option-value pairs
45+
if (len(args) % 2) != 0 {
46+
return errors.New("needs '<option> <value>' to be in pairs")
47+
}
48+
49+
return nil
50+
}
51+
52+
func clusterSetCmdRun(cmd *cobra.Command, args []string) {
53+
options := args[:]
54+
if err := clusterOptionJSONHandler(cmd, options); err != nil {
55+
if GlobalFlag.Verbose {
56+
log.WithError(err).Error("cluster option set failed")
57+
}
58+
failure("Cluster option set failed", err, 1)
59+
} else {
60+
fmt.Printf("Options set successfully \n")
61+
}
62+
}
63+
64+
func clusterOptionJSONHandler(cmd *cobra.Command, options []string) error {
65+
copt := make(map[string]string)
66+
for op, val := range options {
67+
if op%2 == 0 {
68+
copt[val] = options[op+1]
69+
}
70+
}
71+
72+
err := client.ClusterOptionSet(api.ClusterOptionReq{
73+
Options: copt})
74+
if err != nil {
75+
return err
76+
}
77+
return nil
78+
}
79+
80+
var clusterOptionGetCmd = &cobra.Command{
81+
Use: "get",
82+
Short: helpClusterOptionGetCmd,
83+
Args: cobra.RangeArgs(0, 1),
84+
Run: func(cmd *cobra.Command, args []string) {
85+
table := tablewriter.NewWriter(os.Stdout)
86+
opts, err := client.GetClusterOption()
87+
if err != nil {
88+
if GlobalFlag.Verbose {
89+
log.WithError(err).Error("error getting cluster options")
90+
}
91+
failure("Error getting cluster options", err, 1)
92+
}
93+
table.SetHeader([]string{"Name", "Modified", "Value", "Default Value"})
94+
table.SetAlignment(tablewriter.ALIGN_LEFT)
95+
for _, opt := range opts {
96+
table.Append([]string{opt.Key, formatBoolYesNo(opt.Modified), opt.Value, opt.DefaultValue})
97+
}
98+
table.Render()
99+
},
100+
}

‎glustercli/cmd/glustershd.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,6 @@ var selfHealSplitBrainCmd = &cobra.Command{
166166
if err != nil {
167167
failure(fmt.Sprintf("Failed to resolve split-brain for volume %s\n", volname), err, 1)
168168
}
169-
fmt.Printf("Split Brain Resolution successfull on volume %s \n", volname)
169+
fmt.Printf("Split Brain Resolution successful on volume %s \n", volname)
170170
},
171171
}

‎glustercli/cmd/root.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ func addSubCommands(rootCmd *cobra.Command) {
5454
rootCmd.AddCommand(georepCmd)
5555
rootCmd.AddCommand(snapshotCmd)
5656
rootCmd.AddCommand(volumeCmd)
57+
rootCmd.AddCommand(clusterCmd)
5758
}
5859

5960
// GlustercliOption will have all global flags set during run time

‎glustercli/cmd/volume.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -279,6 +279,8 @@ var volumeGetCmd = &cobra.Command{
279279
}
280280
}
281281
table.Render()
282+
fmt.Println("")
283+
282284
}
283285
},
284286
}

‎glusterd2/brickmux/demultiplex.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ func Demultiplex(b brick.Brickinfo) error {
5656
if rsp.OpRet != 0 {
5757
return fmt.Errorf("detach brick RPC request failed; OpRet = %d", rsp.OpRet)
5858
}
59-
log.WithField("brick", b.String()).Debug("detach request succeded with result")
59+
log.WithField("brick", b.String()).Debug("detach request succeeded with result")
6060

6161
// TODO: Find an alternative to substitute the sleep.
6262
// There might be some changes on glusterfsd side related to socket

‎glusterd2/brickmux/reconcile.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ func Reconcile() error {
6666
// do nothing, fallback to starting a separate process
6767
log.WithField("brick", b.String()).Warn(err)
6868
default:
69-
// log and move on, do not exit; this behaviour can be changed later if necessarry
69+
// log and move on, do not exit; this behaviour can be changed later if necessary
7070
log.WithField("brick", b.String()).WithError(err).Error("brickmux.Multiplex failed")
7171
continue
7272
}

‎glusterd2/commands/snapshot/snapshot-clone.go‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -39,16 +39,17 @@ func undoSnapshotClone(c transaction.TxnCtx) error {
3939
return nil
4040
}
4141
func undoStoreSnapshotClone(c transaction.TxnCtx) error {
42-
var vol volume.Volinfo
43-
if err := c.Get("volinfo", &vol); err != nil {
44-
return err
45-
}
42+
var (
43+
vol volume.Volinfo
44+
err error
45+
)
4646

47-
if err := volume.DeleteVolume(vol.Name); err != nil {
47+
if err = c.Get("volinfo", &vol); err != nil {
4848
return err
4949
}
5050

51-
return nil
51+
err = volume.DeleteVolume(vol.Name)
52+
return err
5253
}
5354

5455
func storeSnapshotClone(c transaction.TxnCtx) error {

‎glusterd2/commands/snapshot/snapshot-delete.go‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -83,16 +83,16 @@ func snapshotDelete(c transaction.TxnCtx) error {
8383

8484
func snapshotDeleteStore(c transaction.TxnCtx) error {
8585

86-
var snapinfo snapshot.Snapinfo
87-
if err := c.Get("snapinfo", &snapinfo); err != nil {
88-
return err
89-
}
90-
91-
if err := snapshot.DeleteSnapshot(&snapinfo); err != nil {
86+
var (
87+
snapinfo snapshot.Snapinfo
88+
err error
89+
)
90+
if err = c.Get("snapinfo", &snapinfo); err != nil {
9291
return err
9392
}
9493

95-
return nil
94+
err = snapshot.DeleteSnapshot(&snapinfo)
95+
return err
9696
}
9797

9898
/*

0 commit comments

Comments
 (0)