Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DataNode ¶
type DataNode struct {
DataPath string
MetaPath string
ActPath string
// Assigned after each format.
// When DataNode first starts, it will perform a
// handshake with NameNode. During this process
// NamespaceID will be verified. (also software version
// as described in paper, but I omit it)
NamespaceID int
// Persistent to disk, generated when DataNode first
// registers with NameNode
StorageID string
HostName string // e.g. thumm02
IP string
Port string
Addr string
/* Each block has tow files on DataNode:
* 1. metadata file
* 2. actual data file
* Since DataNode will be requested with block id to
* retrieve data on disk, there is no need to store
* these meta information in memory.
* When a DataNode starts, it performs the following
* actions:
* 1. perform handshake with NameNode, verify NamespaceID.
* NamespaceID will be reassigned after each format action.
* 2. register with NameNode, DataNode get a unique
* StorageID, which is persistent to disk. So if
* the DataNode restart with different IP, it will
* still be able to work.
* 3. send a block report to NameNode. Report each block's
* blockID, generation stamp(here I use timestamp instead)
* and block length. block report then is send periodically
* to NameNode. (every hour as described in paper)
* 4. start sending heartbeats to NameNode. (every 3 seconds in paper)
* Each heartbeat carries: total storage capacity, fraction
* of storage in use, # of data transfer in progress.
* Also report corrupt block to NameNode.
* DataNode is considered died if NameNode hasn't received
* its heartbeat for a very long time. (10 mins in paper)
*/
// IDList []string
IDToMetaData map[string]utils.MetaData
// contains filtered or unexported fields
}
DataNode contains block names and block name to metadata mapping IDToMetaData will be persistent on disk IDList is restored as IDToMetaData.keys()
func NewDataNode ¶
func NewDataNode() *DataNode
NewDataNode retrieve NamespaceID and StorageID on disk (if exist)
func (*DataNode) CalMeanVarMap ¶
CalMeanVarMap calculates mean and variance for this segment
func (*DataNode) RequestBlk ¶
func (d *DataNode) RequestBlk(args *RequestBlkArgs, reply *utils.BlkData) error
RequestBlk will read two files on disk to construct meta data and actual perspectively
func (*DataNode) Run ¶
func (d *DataNode) Run()
Run first perform handshake with NameNode, then register with NameNode to get storage id
func (*DataNode) SendBlk ¶
func (d *DataNode) SendBlk(args *utils.BlkData, reply *SendBlkReply) error
SendBlk is called by client Upon receiving the block data [BlkID, Data, Checksum], datanode will store the meta data in metadata path (data/id2meta) the actual data will be stored in actual data path (data/actdata) for each block, these two files have the same file name: BlkID which is of format: filename-index-timestamp-random datanode will also update its in memory map: IDToMetaData
type RequestBlkArgs ¶
type RequestBlkArgs struct {
BlkID string
}
RequestBlkArgs is used by client to request a block
type SendBlkReply ¶
type SendBlkReply struct {
Status bool
}
SendBlkReply contains status, the argument is BlkData