datanode

package
v0.0.0-...-1a12c4e Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 20, 2020 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type DataNode

type DataNode struct {
	DataPath string
	MetaPath string
	ActPath  string
	// Assigned after each format.
	// When DataNode first starts, it will perform a
	// handshake with NameNode. During this process
	// NamespaceID will be verified. (also software version
	// as described in paper, but I omit it)
	NamespaceID int
	// Persistent to disk, generated when DataNode first
	// registers with NameNode
	StorageID string
	HostName  string // e.g. thumm02
	IP        string
	Port      string
	Addr      string
	/* Each block has tow files on DataNode:
	 * 1. metadata file
	 * 2. actual data file
	 * Since DataNode will be requested with block id to
	 * retrieve data on disk, there is no need to store
	 * these meta information in memory.
	 * When a DataNode starts, it performs the following
	 * actions:
	 * 1. perform handshake with NameNode, verify NamespaceID.
	 *    NamespaceID will be reassigned after each format action.
	 * 2. register with NameNode, DataNode get a unique
	 *    StorageID, which is persistent to disk. So if
	 *    the DataNode restart with different IP, it will
	 *    still be able to work.
	 * 3. send a block report to NameNode. Report each block's
	 *    blockID, generation stamp(here I use timestamp instead)
	 *	  and block length. block report then is send periodically
	 *    to NameNode. (every hour as described in paper)
	 * 4. start sending heartbeats to NameNode. (every 3 seconds in paper)
	 *    Each heartbeat carries: total storage capacity, fraction
	 *    of storage in use, # of data transfer in progress.
	 *    Also report corrupt block to NameNode.
	 *    DataNode is considered died if NameNode hasn't received
	 *    its heartbeat for a very long time. (10 mins in paper)
	 */
	// IDList       []string
	IDToMetaData map[string]utils.MetaData
	// contains filtered or unexported fields
}

DataNode contains block names and block name to metadata mapping IDToMetaData will be persistent on disk IDList is restored as IDToMetaData.keys()

func NewDataNode

func NewDataNode() *DataNode

NewDataNode retrieve NamespaceID and StorageID on disk (if exist)

func (*DataNode) CalMeanVarMap

func (d *DataNode) CalMeanVarMap(args *utils.CalMVArgs, reply *utils.CalMVReply) error

CalMeanVarMap calculates mean and variance for this segment

func (*DataNode) RequestBlk

func (d *DataNode) RequestBlk(args *RequestBlkArgs, reply *utils.BlkData) error

RequestBlk will read two files on disk to construct meta data and actual perspectively

func (*DataNode) Run

func (d *DataNode) Run()

Run first perform handshake with NameNode, then register with NameNode to get storage id

func (*DataNode) SendBlk

func (d *DataNode) SendBlk(args *utils.BlkData, reply *SendBlkReply) error

SendBlk is called by client Upon receiving the block data [BlkID, Data, Checksum], datanode will store the meta data in metadata path (data/id2meta) the actual data will be stored in actual data path (data/actdata) for each block, these two files have the same file name: BlkID which is of format: filename-index-timestamp-random datanode will also update its in memory map: IDToMetaData

type RequestBlkArgs

type RequestBlkArgs struct {
	BlkID string
}

RequestBlkArgs is used by client to request a block

type SendBlkReply

type SendBlkReply struct {
	Status bool
}

SendBlkReply contains status, the argument is BlkData

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL