Syncing on Ethereum - From Consensus header to skeleton chain to the backfilling of Blocks
The entire complete process on how blocks are downloaded locally on GETH
So I’m at the end stage of my journey on the go-ethereum codebase, and pretty much the last part I left out was the syncing process. I mean, I thought about how hard it can be, right? Just request and download the required blocks from the peers and sync until the genesis, but it turns out the whole process is one of the most insane and complicated processes in the whole of the geth codebase. So in this post we’d be going deep into the syncing process at the source code level. Also note, I won’t be going much into the snap sync but rather the full sync.
To give a rough overview of the whole process:
Usually I’d do the mapping from the different state objects, but doing that would make it more complicated. Taking a look at the downloader folder in the codebase under eth/
Honestly, finding an entry point to start the explanation is a bit hard, so I guess we’d go from the least logic to the most logic, starting with resultStore.go, now it may not make sense what the actual use of these is and what is happening at the moment, but just after a while when we come to the actual skeleton, backfiller, and downloader, the whole thing starts to make sense. So understand it for what it is, and soon it’ll start making sense.
This whole process actually starts from the “skeleton” section; both the resultStore and the queue section below are like prerequisites or the fundamentals to understand the whole thing.
resultStore and fetchResults
resultStore is the main structure that maintains fetchResult, tracking their download-progress and delivering the finished results.
1
2
3
4
5
6
7
8
9
10
11
// fetchResult is a struct collecting partial results from data fetchers until
// all outstanding pieces complete and the result as a whole can be processed.
type fetchResult struct {
pending atomic.Int32 // Flag telling what deliveries are outstanding
Header *types.Header
Uncles []*types.Header
Transactions types.Transactions
Receipts rlp.RawValue
Withdrawals types.Withdrawals
}
This is the final struct that is used to build a block. From the comment, we can see that data from each fetcher, e.g., the header fetcher, block body fetcher after fetching the results, stores it in the fetchResult for a particular block which is then finally used to build a block.
fetchResult has different methods with SetBodyDone(), SetReceiptsDone() and importantly AllDone(), AllDone() returns true if both the body and receipts are done.
This is the basic context needed to understand resultStore.go, now we’ll go into detail about the whole resultStore thing.
1
2
3
4
5
6
type resultStore struct {
items []*fetchResult // Downloaded but not yet delivered fetch results
resultOffset uint64
indexIncomplete atomic.Int32
throttleThreshold uint64
}
Methods of the struct:
Prepare(offset) is what is used to initialize the resultStore, it takes in the offset value and updates the resultOffset. So this offset of the first block in the resultStore.
AddFetch(header) adds a new header to the resultStore, first a new fetchResult is created for the particular header with empty body. Then the fetchResult is added to the items array. The index of the the fetchResult being ‘header-resultOffset’
GetCompleted() returns the completed items, meaning fetchResults with their AllDone() returning true, remember that in the items it’s “*fetchResult” , it is a pointer and is no directly manipulation of the fetchResult in the resultStore.go
ThrottleThreshold is used to the set the max items in the resultStore, when adding a new header, throttled will be true if the queue is at it’s capacity and stale if the block is older than the offset block.
Example:
Initializing the store with block 100, setting resultOffset = 100. If the header with block number 101 then the offset being 1 Transactions and withdrawals are added to the fetchResults then calling GetCompleted() will return the fetchResults which are completed.
queue
The queue acts as the main coordinator of the fetchResult array. This is where the scheduling of headers takes place; reservebodies assigns which headers the peer should fetch, and then finally delivers the results to the fetchResult for the particular header.
Schedule(headers[], hashes[]), accepts a set of headers which are needed to be added to the download queue. It loops through the headers, verifies if the ordering is right, then adds it to the blockTaskPool[hash] = header and blockTaskQueue. For snap syncing for the receipt retrieval, the header is added to the receiptTaskPool and receiptTaskQueue.
Next is the ReserveBodies and the ReserveReceipts function, which creates a new fetchResult and reserve which peers to download how many headers.
1
func (q *queue) ReserveBodies(p *peerConnection, count int) (*fetchRequest, bool, bool) {
It takes in a peer connection and count para being the number of headers to assign for the particular peer.
ReserveBodies inside calls reserveHeaders() function which loops through each header until ‘count’ , calls resultStore.AddFetch(header) which again creates a new fetchresult. Then a new fetch request is created with the peer and the requested headers to send.
1
2
3
4
5
request := &fetchRequest{
Peer: p,
Headers: send,
Time: time.Now(),
}
Same thing for the ReserveReceipts() too where in it assigns peers for the receipts fetching too.
Next up the DeliverBodies() and DeliverReceipts()
1
func (q *queue) DeliverBodies(id string, hashes eth.BlockBodyHashes, bodies []eth.BlockBody) (int, error) {
This function accepts the block body and assign it to a particular request, it first validates the block body, then defines the reconstruct function which modifies the fetchResult,
1
2
3
4
5
6
reconstruct := func(index int, result *fetchResult) {
result.Transactions = txLists[index]
result.Uncles = uncleLists[index]
result.Withdrawals = withdrawalLists[index]
result.SetBodyDone()
}
Which again calls deliver() function which fetches the resultCache for the particular header from the resultStore:
1
if res, stale, err := q.resultCache.GetDeliverySlot(header.Number.Uint64()); err == nil && !stale {
Then calls the reconstruction function modifying the fetchResult hence appending the transactions , withdrawals and the uncles, in the reconstruct we can see that the SetBodyDone is being called.
For receipts same thing happens instead of body where inturn SetReceiptsDone() is called.
So for a fetchResult if both the body and the receipts is done, it can be marked as a complete or calling AllDone() will return true.
So basically queue is like a layer before resultStore.
skeleton
Now let’s get to the part where the actual sync process happens. The logic is that after the merge, the whole syncing process is of a new design. There is a temporary header chain named the skeleton where the headers are downloaded in reverse. The consensus chain tells Geth that this is the particular head now, and Geth starts downloading the headers in reverse. Now, in this reverse process, first the headers are downloaded and stored in a temporary chain called the skeleton, and then once the chain is linked, the actual backfilling process happens, meaning the contents of the block are downloaded for the particular headers.
So basically a new head is announced: -> a skeleton chain is formed where the headers are downloaded until a local block is known . -> then later the contents of the headers are backfilled meaning the whole block is formed -> Then the insert method in the blockchain is called inserting and executing the blocks.
This is the entire process of syncing on ethereum!
Now let’s get to the code.
We’ll directly start with the sync(header) function in skeleton.go and later see how it will come to that.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
// sync is the internal version of Sync that executes a single sync cycle, either
// until some termination condition is reached, or until the current cycle merges
// with a previously aborted run.
func (s *skeleton) sync(head *types.Header) (*types.Header, error) {
// If we're continuing a previous merge interrupt, just access the existing
// old state without initing from disk.
if head == nil {
head = rawdb.ReadSkeletonHeader(s.db, s.progress.Subchains[0].Head)
} else {
// Otherwise, initialize the sync, trimming and previous leftovers until
// we're consistent with the newly requested chain head
s.initSync(head)
}
// Create the scratch space to fill with concurrently downloaded headers
s.scratchSpace = make([]*types.Header, scratchHeaders)
defer func() { s.scratchSpace = nil }() // don't hold on to references after sync
s.scratchOwners = make([]string, scratchHeaders/requestHeaders)
defer func() { s.scratchOwners = nil }() // don't hold on to references after sync
s.scratchHead = s.progress.Subchains[0].Tail - 1 // tail must not be 0!
// If the sync is already done, resume the backfiller. When the loop stops,
// terminate the backfiller too.
linked := len(s.progress.Subchains) == 1 && s.linked(s.scratchHead, s.progress.Subchains[0].Next)
if linked {
s.filler.resume()
}
defer func() {
// The filler needs to be suspended, but since it can block for a while
// when there are many blocks queued up for full-sync importing, run it
// on a separate goroutine and consume head messages that need instant
// replies.
done := make(chan struct{})
go func() {
defer close(done)
filled := s.filler.suspend()
if filled == nil {
log.Warn("Latest filled block is not available")
return
}
// If something was filled, try to delete stale sync helpers. If
// unsuccessful, warn the user, but not much else we can do (it's
// a programming error, just let users report an issue and don't
// choke in the meantime).
if err := s.cleanStales(filled); err != nil {
log.Error("Failed to clean stale beacon headers", "err", err)
}
}()
// Wait for the suspend to finish, consuming head events in the meantime
// and dropping them on the floor.
for {
select {
case <-done:
return
case event := <-s.headEvents:
event.errc <- errors.New("beacon syncer reorging")
}
}
}()
// Create a set of unique channels for this sync cycle. We need these to be
// ephemeral so a data race doesn't accidentally deliver something stale on
// a persistent channel across syncs (yup, this happened)
var (
requestFails = make(chan *headerRequest)
responses = make(chan *headerResponse)
)
cancel := make(chan struct{})
defer close(cancel)
log.Debug("Starting reverse header sync cycle", "head", head.Number, "hash", head.Hash(), "cont", s.scratchHead)
// Whether sync completed or not, disregard any future packets
defer func() {
log.Debug("Terminating reverse header sync cycle", "head", head.Number, "hash", head.Hash(), "cont", s.scratchHead)
s.requests = make(map[uint64]*headerRequest)
}()
// Start tracking idle peers for task assignments
peering := make(chan *peeringEvent, 64) // arbitrary buffer, just some burst protection
peeringSub := s.peers.SubscribeEvents(peering)
defer peeringSub.Unsubscribe()
s.idles = make(map[string]*peerConnection)
for _, peer := range s.peers.AllPeers() {
s.idles[peer.id] = peer
}
// Notify any tester listening for startup events
if s.syncStarting != nil {
s.syncStarting()
}
for {
// Something happened, try to assign new tasks to any idle peers
if !linked {
s.assignTasks(responses, requestFails, cancel)
}
// Wait for something to happen
select {
case event := <-peering:
// A peer joined or left, the tasks queue and allocations need to be
// checked for potential assignment or reassignment
peerid := event.peer.id
if event.join {
log.Debug("Joining skeleton peer", "id", peerid)
s.idles[peerid] = event.peer
} else {
log.Debug("Leaving skeleton peer", "id", peerid)
s.revertRequests(peerid)
delete(s.idles, peerid)
}
case errc := <-s.terminate:
errc <- nil
return nil, errTerminated
case event := <-s.headEvents:
// New head was announced, try to integrate it. If successful, nothing
// needs to be done as the head simply extended the last range. For now
// we don't seamlessly integrate reorgs to keep things simple. If the
// network starts doing many mini reorgs, it might be worthwhile handling
// a limited depth without an error.
if err := s.processNewHead(event.header, event.final); err != nil {
// If a reorg is needed, and we're forcing the new head, signal
// the syncer to tear down and start over. Otherwise, drop the
// non-force reorg.
if event.force {
event.errc <- nil // forced head reorg accepted
log.Info("Restarting sync cycle", "reason", err)
return event.header, errSyncReorged
}
event.errc <- err
continue
}
event.errc <- nil // head extension accepted
// New head was integrated into the skeleton chain. If the backfiller
// is still running, it will pick it up. If it already terminated,
// a new cycle needs to be spun up.
if linked {
if len(s.progress.Subchains) == 1 && s.linked(s.scratchHead, s.progress.Subchains[0].Next) {
// The skeleton chain has been extended and is still linked with the local
// chain, try to re-schedule the backfiller if it's already terminated.
s.filler.resume()
} else {
// The skeleton chain is no longer linked to the local chain for some reason
// (e.g. debug_setHead was used to trim the local chain). Re-schedule the
// skeleton sync to fill the chain gap.
log.Warn("Local chain has been trimmed", "tailnumber", s.scratchHead, "tailhash", s.progress.Subchains[0].Next)
return nil, errSyncTrimmed
}
}
case req := <-requestFails:
s.revertRequest(req)
case res := <-responses:
// Process the batch of headers. If though processing we managed to
// link the current subchain to a previously downloaded one, abort the
// sync and restart with the merged subchains.
//
// If we managed to link to the existing local chain or genesis block,
// abort sync altogether.
linked, merged := s.processResponse(res)
if linked {
log.Debug("Beacon sync linked to local chain")
return nil, errSyncLinked
}
if merged {
log.Debug("Beacon sync merged subchains")
return nil, errSyncMerged
}
// We still have work to do, loop and repeat
}
}
}
So the sync accepts the current head block, then first the initSync() function is called:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
func (s *skeleton) initSync(head *types.Header) {
// Extract the head number, we'll need it all over
number := head.Number.Uint64()
s.progress = &skeletonProgress{
Subchains: []*subchain{
{
Head: number,
Tail: number,
Next: head.ParentHash,
},
},
}
Here a new subchain is created or a new skeleton header chain is created:
Suppose the newly received head is 9999, so Subchains[0].Head = 9999 Subchains[0].Tail = 9999 Subchains[0].Next = Hash(9998)
Now a scratchSpace is created which is an in-memory cache which holds the headers,
1
s.scratchSpace = make([]*types.Header, scratchHeaders)
Then a scratchOwners array:
1
2
3
4
s.scratchOwners = make([]string, scratchHeaders/requestHeaders)
const scratchHeaders = 131072
const requestHeaders = 512
1
s.scratchHead = s.progress.Subchains[0].Tail - 1 // tail must not be 0!
ScratchHead being from the example 9998 and the first position in the scratchSpace
So per peer at a time max 512 headers are requested. Scratch owners is basically the index for each peer
1
2
3
4
5
6
// If the sync is already done, resume the backfiller. When the loop stops,
// terminate the backfiller too.
linked := len(s.progress.Subchains) == 1 && s.linked(s.scratchHead, s.progress.Subchains[0].Next)
if linked {
s.filler.resume()
}
This is pretty much the most important condition in the function linked() function basically checks if the header chain is linked with the local chain, meaning it checks if the skeleton chain is linked with the localChain s.linked(9998, hash(9998)) if true means the block is available locally and the backfilling can be started for the block 9999 which is the newly announced block.
1
2
3
4
5
6
for {
// Something happened, try to assign new tasks to any idle peers
if !linked {
s.assignTasks(responses, requestFails, cancel)
}
// Wait for something to happen
So basically if the chain is not linked meaning there are missing blocks which need to be downloaded, the assignTasks function assigns the download job for each peers and the number of headers to be downloaded, it passes the responses channel to receive the sent events from the peers.
1
func (s *skeleton) assignTasks(success chan *headerResponse, fail chan *headerRequest, cancel chan struct{}) {
Initially the sorting of the peers happen, and then the assigning process starts:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
// assignTasks attempts to match idle peers to pending header retrievals.
func (s *skeleton) assignTasks(success chan *headerResponse, fail chan *headerRequest, cancel chan struct{}) {
// Sort the peers by download capacity to use faster ones if many available
idlers := &peerCapacitySort{
peers: make([]*peerConnection, 0, len(s.idles)),
caps: make([]int, 0, len(s.idles)),
}
targetTTL := s.peers.rates.TargetTimeout()
for _, peer := range s.idles {
idlers.peers = append(idlers.peers, peer)
idlers.caps = append(idlers.caps, s.peers.rates.Capacity(peer.id, eth.BlockHeadersMsg, targetTTL))
}
if len(idlers.peers) == 0 {
return
}
sort.Sort(idlers)
// Find header regions not yet downloading and fill them
for task, owner := range s.scratchOwners {
// If we're out of idle peers, stop assigning tasks
if len(idlers.peers) == 0 {
return
}
// Skip any tasks already filling
if owner != "" {
continue
}
// If we've reached the genesis, stop assigning tasks
if uint64(task*requestHeaders) >= s.scratchHead {
return
}
// Found a task and have peers available, assign it
idle := idlers.peers[0]
idlers.peers = idlers.peers[1:]
idlers.caps = idlers.caps[1:]
// Matched a pending task to an idle peer, allocate a unique request id
var reqid uint64
for {
reqid = uint64(rand.Int63())
if reqid == 0 {
continue
}
if _, ok := s.requests[reqid]; ok {
continue
}
break
}
// Generate the network query and send it to the peer
req := &headerRequest{
peer: idle.id,
id: reqid,
deliver: success,
revert: fail,
cancel: cancel,
stale: make(chan struct{}),
head: s.scratchHead - uint64(task*requestHeaders),
}
s.requests[reqid] = req
delete(s.idles, idle.id)
// Generate the network query and send it to the peer
go s.executeTask(idle, req)
// Inject the request into the task to block further assignments
s.scratchOwners[task] = idle.id
}
}
ScratchOwners has 256 entries based on the division and each entry represents one possible batch of the 512 headers.
In each iteration it takes peer from the idlePeer array and assigns the batch of headers to be downloaded , the loop breaks when there are no idle peers left.
1
2
3
4
5
6
7
8
9
req := &headerRequest{
peer: idle.id,
id: reqid,
deliver: success,
revert: fail,
cancel: cancel,
stale: make(chan struct{}),
head: s.scratchHead - uint64(task*requestHeaders),
}
1
s.scratchHead - uint64(task*requestHeaders),
This is particularly the most important line, let’s understand it by an example:
ScratchHead = 9999
First idle peer head = 9999 - 0*512 = 9999 next = 9999 - 1*512 = 9487 next = 9999 - 2*512 = 8975
Now these are the heads assigned to each peer to be downloaded, in the next function executeTask is where the actual process happens and the amount of headers to be downloaded being 512, so
8975+512 = 9487 9487+512 = 9999
We can see this matches the headers correctly!
Now going into the executeTask function:
1
func (s *skeleton) executeTask(peer *peerConnection, req *headerRequest) {
1
2
3
4
requestCount := requestHeaders
if req.head < requestHeaders {
requestCount = int(req.head)
}
We can see the number of request count being 512
1
netreq, err := peer.peer.RequestHeadersByNumber(req.head, requestCount, 0, true, resCh)
And now that’s it! , The actual request to the particular peer RequestHeadersByNumber(9487, 512), Which requests the headers from 9487 to 9999
Then we can see the response channel also being passed where the response will be sent.
1
2
3
4
5
6
7
8
9
10
case res := <-resCh:
// Headers successfully retrieved, update the metrics
headers := *res.Res.(*eth.BlockHeadersRequest)
headerReqTimer.Update(time.Since(start))
s.peers.rates.Update(peer.id, eth.BlockHeadersMsg, res.Time, len(headers))
// Cross validate the headers with the requests
switch {
After a few checks , the request is sent to the response channel which was passed during the sync() function:
1
2
3
4
5
case req.deliver <- &headerResponse{
peer: peer,
reqid: req.id,
headers: headers,
}:
Now in the sync() function we can see the loop listening to the response channel:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
case res := <-responses:
// Process the batch of headers. If though processing we managed to
// link the current subchain to a previously downloaded one, abort the
// sync and restart with the merged subchains.
//
// If we managed to link to the existing local chain or genesis block,
// abort sync altogether.
linked, merged := s.processResponse(res)
if linked {
log.Debug("Beacon sync linked to local chain")
return nil, errSyncLinked
}
if merged {
log.Debug("Beacon sync merged subchains")
return nil, errSyncMerged
}
// We still have work to do, loop and repeat
}
Next up is the processResponse() function:
1
func (s *skeleton) processResponse(res *headerResponse) (linked bool, merged bool) {
Initially we can see it adds the peer to the idle array if the response is valid:
1
s.idles[res.peer.id] = res.peer
Then add the received headers to the scratchSpace:
1
2
3
4
// Insert the delivered headers into the scratch space independent of the
// content or continuation; those will be validated in a moment
head := res.headers[0].Number.Uint64()
copy(s.scratchSpace[s.scratchHead-head:], res.headers)
scratchSpace[9999-9487:], 512 headers
So from 512 index ,512 headers are added to the scratchSpace,
1
2
3
4
// If there's still a gap in the head of the scratch space, abort
if s.scratchSpace[0] == nil {
return false, false
}
Here is a waiting process happening , seeing if the first expected batch is sent or not.
1
2
3
4
for s.scratchSpace[0] != nil {
// Next batch of headers available, cross-reference with the subchain
// we are extending and either accept or discard
if s.progress.Subchains[0].Next != s.scratchSpace[0].Hash() {
Next up we can see there is a check going on to see if the headers connect.
1
2
3
4
5
6
7
8
9
for _, header := range s.scratchSpace[:requestHeaders] {
if header != nil { // nil when the genesis is reached
consumed++
rawdb.WriteSkeletonHeader(batch, header)
s.pulled++
s.progress.Subchains[0].Tail--
s.progress.Subchains[0].Next = header.ParentHash
An inner loop again which goes through each individual header and saves it to the skeleton database, Also we can see the tail being reduced showing the reverse process of the skeleton chain.
Then next up the loop breaks if a local known block is reached, which is where the skeleton chain is linked to the local chain.
1
2
3
4
5
6
7
8
9
10
11
12
// If we've reached an existing block in the chain, stop retrieving
// headers. Note, if we want to support light clients with the same
// code we'd need to switch here based on the downloader mode. That
// said, there's no such functionality for now, so don't complicate.
//
// In the case of full sync it would be enough to check for the body,
// but even a full syncing node will generate a receipt once block
// processing is done, so it's just one more "needless" check.
//
// The weird cascading checks are done to minimize the database reads.
linked = s.linked(header.Number.Uint64()-1, header.ParentHash)
If the chain is linked abort the header synchronization process:
1
2
3
4
5
// If the beacon chain was linked to the local chain, completely swap out
// all internal progress and abort header synchronization.
if linked {
// Merge all overlapped subchains beforehand
s.mergeSubchains()
The mergeSubchains is pretty much self explanatory, during the syncing process if something happens to the syncing process the subchain breaks and just get split into multiple subchains.
Now the same thing happens for the next batch of headers:
1
2
// Batch of headers consumed, shift the download window forward
copy(s.scratchSpace, s.scratchSpace[requestHeaders:])
1
2
3
4
5
6
7
8
9
10
if s.pulled == 0 {
log.Info("Beacon sync starting", "left", left)
} else {
eta := float64(time.Since(s.started)) / float64(s.pulled) * float64(left)
log.Info("Syncing beacon headers", "downloaded", s.pulled, "left", left, "eta", common.PrettyDuration(eta))
}
}
return linked, merged
Thus being the end of the processResponse() function and returning whether the chain was linked or not.
Next up we can see the backfilling process but before that like I’ve mentioned in the starting we’d have to see how the whole thing comes down to the sync() function.
The Sync(header, final) function is called in the forkchoiceupdated function which is again called by the beacon client, we can see:
1
if err := api.eth.Downloader().BeaconSync(header, finalized); err != nil {
BeaconSync being:
1
2
3
4
5
// Internally backfilling and state sync is done the same way, but the header
// retrieval and scheduling is replaced.
func (d *Downloader) BeaconSync(head *types.Header, final *types.Header) error {
return d.beaconSync(head, final, true)
}
The Sync function passes the header to the headEvents channel:
1
case s.headEvents <- &headUpdate{header: head, final: final, force: force, errc: errc}:
There are two go routines listening to the headEvents channel:
One initially when a new skeleton chain is defined:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func (s *skeleton) startup() {
// Close a notification channel so anyone sending us events will know if the
// sync loop was torn down for good.
defer close(s.terminated)
// Wait for startup or teardown. This wait might loop a few times if a beacon
// client requests sync head extensions, but not forced reorgs (i.e. they are
// giving us new payloads without setting a starting head initially).
for {
select {
case errc := <-s.terminate:
// No head was announced but Geth is shutting down
errc <- nil
return
case event := <-s.headEvents:
Then in the sync() function itself:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
case event := <-s.headEvents:
// New head was announced, try to integrate it. If successful, nothing
// needs to be done as the head simply extended the last range. For now
// we don't seamlessly integrate reorgs to keep things simple. If the
// network starts doing many mini reorgs, it might be worthwhile handling
// a limited depth without an error.
if err := s.processNewHead(event.header, event.final); err != nil {
// If a reorg is needed, and we're forcing the new head, signal
// the syncer to tear down and start over. Otherwise, drop the
// non-force reorg.
if event.force {
event.errc <- nil // forced head reorg accepted
log.Info("Restarting sync cycle", "reason", err)
return event.header, errSyncReorged
}
event.errc <- err
continue
}
event.errc <- nil // head extension accepted
// New head was integrated into the skeleton chain. If the backfiller
// is still running, it will pick it up. If it already terminated,
// a new cycle needs to be spun up.
if linked {
if len(s.progress.Subchains) == 1 && s.linked(s.scratchHead, s.progress.Subchains[0].Next) {
// The skeleton chain has been extended and is still linked with the local
// chain, try to re-schedule the backfiller if it's already terminated.
s.filler.resume()
} else {
So when a new head is received from the consensus engine the whole syncing starts, also whenever a new head is received the sync() function, the backfilling process starts,
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
case event := <-s.headEvents:
// New head was announced, try to integrate it. If successful, nothing
// needs to be done as the head simply extended the last range. For now
// we don't seamlessly integrate reorgs to keep things simple. If the
// network starts doing many mini reorgs, it might be worthwhile handling
// a limited depth without an error.
if err := s.processNewHead(event.header, event.final); err != nil {
// If a reorg is needed, and we're forcing the new head, signal
// the syncer to tear down and start over. Otherwise, drop the
// non-force reorg.
if event.force {
event.errc <- nil // forced head reorg accepted
log.Info("Restarting sync cycle", "reason", err)
return event.header, errSyncReorged
}
event.errc <- err
continue
}
event.errc <- nil // head extension accepted
// New head was integrated into the skeleton chain. If the backfiller
// is still running, it will pick it up. If it already terminated,
// a new cycle needs to be spun up.
if linked {
if len(s.progress.Subchains) == 1 && s.linked(s.scratchHead, s.progress.Subchains[0].Next) {
// The skeleton chain has been extended and is still linked with the local
// chain, try to re-schedule the backfiller if it's already terminated.
s.filler.resume()
The processNewHead() function checks for reorgs, gaps and stuffs like that then finally and most importantly the backfiller is started by calling:
1
s.filler.resume()
Which is again a new whole section where the whole downloading of the actual block contents of the headers starts! On to the backfiller now.
backfiller
backfiller is the main process that does the filling up of headers; it has two methods, resume() and suspend(), the methods in beaconsync. go, or the beaconBackfiller acts as a middleman between the skeleton and the downloader process.
resume() is used to start the backfiller, which internally calls the synchronise() method in the downloader.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// resume starts the downloader threads for backfilling state and chain data.
func (b *beaconBackfiller) resume()
if err := b.downloader.synchronise(b.started); err != nil {
log.Error("Beacon backfilling failed", "err", err)
return
}
// Synchronization succeeded. Since this happens async, notify the outer
// context to enable transaction propagation.
if b.success != nil {
b.success()
}
log.Debug("Backfilling completed")
}()
log.Debug("Backfilling started")
}
downloader
downloader is what starts the actual backfilling process of the headers, directly onto the synchronise function:
1
2
3
4
// synchronise will select the peer and use it for synchronising. If an empty string is given
// it will use the best peer possible and synchronize if its TD is higher than our own. If any of the
// checks fail an error will be returned. This method is synchronous
func (d *Downloader) synchronise(beaconPing chan struct{}) (err error) {
Few checks and definitions here and there and calls syncToHead()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// syncToHead starts a block synchronization based on the hash chain from
// the specified head hash.
func (d *Downloader) syncToHead() (err error) {
d.mux.Post(StartEvent{})
defer func() {
// reset on error
if err != nil {
d.mux.Post(FailedEvent{err})
} else {
latest := d.blockchain.CurrentHeader()
d.mux.Post(DoneEvent{latest})
}
}()
mode := d.getMode()
log.Debug("Backfilling with the network", "mode", mode)
defer func(start time.Time) {
log.Debug("Synchronisation terminated", "elapsed", common.PrettyDuration(time.Since(start)))
}(time.Now())
// Look up the sync boundaries: the common ancestor and the target block
var latest, pivot, final *types.Header
latest, _, final, err = d.skeleton.Bounds()
if err != nil {
return err
}
syncToHead() first calls in skeleton.Bounds() which returns the head, tail and finalized block in the skeleton header range.
Then we have the pivot header being defined which 64 blocks behind the the latest header:
1
2
3
4
5
6
if latest.Number.Uint64() > uint64(fsMinFullBlocks) {
number := latest.Number.Uint64() - uint64(fsMinFullBlocks)
// Retrieve the pivot header from the skeleton chain segment but
// fallback to local chain if it's not found in skeleton space.
if pivot = d.skeleton.Header(number); pivot == nil {
1
2
3
4
height := latest.Number.Uint64()
origin, err := d.findBeaconAncestor()
chainOffset := origin + 1
Here origin is the common block between the skeleton chain and the local chain.
Next up is “ if mode == ethconfig.SnapSync {“ , which is for snap sync, Just like I mentioned previously we won’t be going deep into the snapsync process.
1
d.queue.Prepare(chainOffset, mode)
Queue is being initialized with the chainOffset being the common ancestor + 1, where geth already has the common ancestor so start from the next block.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
fetchers := []func() error{
func() error { return d.fetchHeaders(origin + 1) }, // Headers are always retrieved
func() error { return d.fetchBodies(chainOffset) }, // Bodies are retrieved during normal and snap sync
func() error { return d.fetchReceipts(chainOffset) }, // Receipts are retrieved during snap sync
func() error { return d.processHeaders(origin + 1) },
}
if mode == ethconfig.SnapSync {
d.pivotLock.Lock()
d.pivotHeader = pivot
d.pivotLock.Unlock()
fetchers = append(fetchers, func() error { return d.processSnapSyncContent() })
} else if mode == ethconfig.FullSync {
fetchers = append(fetchers, func() error { return d.processFullSyncContent() })
}
return d.spawnSync(fetchers)
There are the fetchers which does the fetching of headers from the skeleton chain and updating the localchain, then fetching the block bodies, receipts and updating it on the main blockchain!
1
2
3
// fetchHeaders feeds skeleton headers to the downloader queue for scheduling
// until sync errors or is finished.
func (d *Downloader) fetchHeaders(from uint64) error {
Skipping the few checks that happen initially, a loop starts and then the headers are fetched from the skeleton chain:
1
2
3
4
header := d.skeleton.Header(from)
headers = append(headers, header)
hashes = append(hashes, headers[i].Hash())
from++
1
2
3
4
5
6
log.Trace("Scheduling new beacon headers", "count", len(headers), "from", from-uint64(len(headers)))
select {
case d.headerProcCh <- &headerTask{
headers: headers,
hashes: hashes,
}:
Then it is sent to the headerProcCh channel which is picked up by the processHeaders() function.
Next fetcher is the blockBodies fethcher function which internally calls concurrentFetch()
1
func (d *Downloader) concurrentFetch(queue typedQueue) error {
Now this is exactly where we need to understand the queue properly which I’ve explained throughly in the beginning of the post, so make sure to understand that.
1
responses := make(chan *eth.Response)
First a responses channel is created.
1
2
3
4
5
for _, peer := range d.peers.AllPeers() {
pending, stale := pending[peer.id], stales[peer.id]
if pending == nil && stale == nil {
idles = append(idles, peer)
caps = append(caps, queue.capacity(peer, time.Second))
A peer queue is created.
Then a loop starts for each peer:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
for _, peer := range idles {
// Short circuit if throttling activated or there are no more
// queued tasks to be retrieved
if throttled {
break
}
if queued := queue.pending(); queued == 0 {
break
}
// Reserve a chunk of fetches for a peer. A nil can mean either that
// no more headers are available, or that the peer is known not to
// have them.
request, _, throttle := queue.reserve(peer, queue.capacity(peer, d.peers.rates.TargetRoundTrip()))
By calling queue.reserve particular header range is assigned for the particular peer.
1
req, err := queue.request(peer, request, responses)
Then the request is called which calls the RequestBodies() for the peer:
1
2
3
4
5
6
7
8
9
10
11
12
13
// request is responsible for converting a generic fetch request into a body
// one and sending it to the remote peer for fulfillment.
func (q *bodyQueue) request(peer *peerConnection, req *fetchRequest, resCh chan *eth.Response) (*eth.Request, error) {
peer.log.Trace("Requesting new batch of bodies", "count", len(req.Headers), "from", req.Headers[0].Number)
if q.bodyFetchHook != nil {
q.bodyFetchHook(req.Headers)
}
hashes := make([]common.Hash, 0, len(req.Headers))
for _, header := range req.Headers {
hashes = append(hashes, header.Hash())
}
return peer.peer.RequestBodies(hashes, resCh)
}
1
pending[peer.id] = req
The request is assigned to the particular peer.
Then the peer sends the response to the responses channel:
1
2
3
4
5
6
7
8
case res := <-responses:
if peer := d.peers.Peer(res.Req.Peer); peer != nil {
// Deliver the received chunk of data and check chain validity
accepted, err := queue.deliver(peer, res)
if errors.Is(err, errInvalidChain) {
return err
}
Which calls the queue deliverbody method which updates the fetchResult and adding the bodies.
Next up is the processHeaders() fetcher which listens to the headerProcCh channel:
1
2
3
4
5
6
for {
select {
case <-d.cancelCh:
return errCanceled
case task := <-d.headerProcCh:
Where the headers are inserted into the chain:
1
2
3
4
5
6
7
8
if mode == ethconfig.SnapSync && cutoff != 0 {
if n, err := d.blockchain.InsertHeadersBeforeCutoff(chunkHeaders[:cutoff]); err != nil {
log.Warn("Failed to insert ancient header chain", "number", chunkHeaders[n].Number, "hash", chunkHashes[n], "parent", chunkHeaders[n].ParentHash, "err", err)
return fmt.Errorf("%w: %v", errInvalidChain, err)
}
log.Debug("Inserted headers before cutoff", "number", chunkHeaders[cutoff-1].Number, "hash", chunkHashes[cutoff-1])
}
Then next fetcher is the processSnapSyncContent() and the processSnapSyncContent() based on full or pivot sync.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
func (d *Downloader) processFullSyncContent() error {
for {
results := d.queue.Results(true)
if len(results) == 0 {
return nil
}
if d.chainInsertHook != nil {
d.chainInsertHook(results)
}
if err := d.importBlockResults(results); err != nil {
return err
}
}
}
The results are fetched from the queue. Then importBlockResults does the block creation and the insertion part.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
func (d *Downloader) importBlockResults(results []*fetchResult) error {
// Check for any early termination requests
if len(results) == 0 {
return nil
}
select {
case <-d.quitCh:
return errCancelContentProcessing
default:
}
// Retrieve a batch of results to import
first, last := results[0].Header, results[len(results)-1].Header
log.Debug("Inserting downloaded chain", "items", len(results),
"firstnum", first.Number, "firsthash", first.Hash(),
"lastnum", last.Number, "lasthash", last.Hash(),
)
blocks := make([]*types.Block, len(results))
for i, result := range results {
blocks[i] = types.NewBlockWithHeader(result.Header).WithBody(result.body())
}
We can see a new block is created.
Then volia!
1
if index, err := d.blockchain.InsertChain(blocks); err != nil {
Thus the blocks are inserted into the main blockchain.
I mean, oh wow, that was indeed a long mapping and hella lot of writing. The entire thing is one of the most complex parts of the Go Ethereum codebase; the number of Go routines running at a particular point in time is just insane tbh.