Multiple Queue improvements: LevelDB Wait on empty, shutdown empty shadow level queue, reduce goroutines etc (#15693)

* move shutdownfns, terminatefns and hammerfns out of separate goroutines

Coalesce the shutdownfns etc into a list of functions that get run at shutdown
rather then have them run at goroutines blocked on selects.

This may help reduce the background select/poll load in certain
configurations.

* The LevelDB queues can actually wait on empty instead of polling

Slight refactor to cause leveldb queues to wait on empty instead of polling.

* Shutdown the shadow level queue once it is empty

* Remove bytefifo additional goroutine for readToChan as it can just be run in run

* Remove additional removeWorkers goroutine for workers

* Simplify the AtShutdown and AtTerminate functions and add Channel Flusher

* Add shutdown flusher to CUQ

* move persistable channel shutdown stuff to Shutdown Fn

* Ensure that UPCQ has the correct config

* handle shutdown during the flushing

* reduce risk of race between zeroBoost and addWorkers

* prevent double shutdown

Signed-off-by: Andrew Thornton <art27@cantab.net>

modules/queue/queue.go

@@ -57,7 +57,7 @@ type Named interface {
 // Queues will handle their own contents in the Run method
 type Queue interface {
 	Flushable
-	Run(atShutdown, atTerminate func(context.Context, func()))
+	Run(atShutdown, atTerminate func(func()))
 	Push(Data) error
 }
 
@@ -74,7 +74,7 @@ type DummyQueue struct {
 }
 
 // Run does nothing
-func (*DummyQueue) Run(_, _ func(context.Context, func())) {}
+func (*DummyQueue) Run(_, _ func(func())) {}
 
 // Push fakes a push of data to the queue
 func (*DummyQueue) Push(Data) error {
@@ -122,7 +122,7 @@ type Immediate struct {
 }
 
 // Run does nothing
-func (*Immediate) Run(_, _ func(context.Context, func())) {}
+func (*Immediate) Run(_, _ func(func())) {}
 
 // Push fakes a push of data to the queue
 func (q *Immediate) Push(data Data) error {