commit 14a9c290d1fa6461dbbd207f18ae5a1023563cdb
Author: Victor Lowther <victor.lowther@gmail.com>
Date: Tue Aug 18 10:07:32 2020 -0500
fix(revalidate): Fix data race when validating content bundles.
So, it turns out we were still falling through to the old revalidate
everything code after processing a content bundle change list. Not
only was this hiding a few bugs with the new code, it caused spurious
data races in the parallel revalidate everything code.
Rework changes to make content bundle changes validate as intended and
thereby avoid the spurious data race.
M backend/dataTracker.go
M backend/profiles_test.go
M backend/utils.go
commit 69b0dc6cce58a981779bb28de5fa7b604e99b06b
Author: Victor Lowther <victor.lowther@gmail.com>
Date: Mon Aug 17 11:57:18 2020 -0500
perf(startup): Parallelize object processing on startup
This adds parallelism in two places at system startup.
The first is when bulk unmarshalling data from the WAL. The
max batch size has been increased from 4 megs to 32, and actual
unmarshalling load is equally divided among the CPUs present on the
system if the total amount of data being unmarshalled passes a certain
threshold.
The second is when validating data when building the datatracker. The
data load for parallelization is divided across all CPUs in the same fashion
as when being bulk unmarshalled, unless the objects in question are
backed by a RawModel -- trying to parallelize operations on them resulted
in concurrent map access panics.
M backend/dataTracker.go
M datastack/index/index.go
M datastack/stack.go
commit 3757b23649a227e8d80ad90912accc50dce278ce
Author: Victor Lowther <victor.lowther@gmail.com>
Date: Mon Aug 17 11:00:52 2020 -0500
perf(startup): Cache sorted indexes.
This adds functionality to the Index layer to recognize when we are
asking for a sorted version of a base index, and caches the result
for later use if we are. The cache will be invalidated if anything
is added, removed, or modified in the base layer of the index.
To enable this to work properly, Names have been added to all Makers
where applicable, to give each index a set of unique keys to work with.
M backend/bootenv.go
M backend/catalog_items.go
M backend/contexts.go
M backend/endpoints.go
M backend/jobs.go
M backend/lease.go
M backend/machines.go
M backend/param.go
M backend/pluginProviders.go
M backend/plugins.go
M backend/pools.go
M backend/profiles.go
M backend/raw_model.go
M backend/reservation.go
M backend/roles.go
M backend/stage.go
M backend/subnet.go
M backend/task.go
M backend/template.go
M backend/tenants.go
M backend/user.go
M backend/version_sets.go
M backend/workflow.go
M datastack/index/index.go
M datastack/index/index_test.go
commit f9ace71745a0d1eff292174745408991f2e0b98c
Author: Victor Lowther <victor.lowther@gmail.com>
Date: Fri Aug 14 13:32:55 2020 -0500
Perf enchancements to machine processing during startup
M backend/bootenv.go
M backend/dataTracker.go
M backend/machines.go
commit a52e38262335c33965eb4d99e793e770ba65beb2
Author: Victor Lowther <victor.lowther@gmail.com>
Date: Fri Aug 14 09:40:32 2020 -0500
feat(walTool): Add more debugging to dr-waltool.
M backend/dataTracker.go
M cmds/dr-waltool/waltool.go
M datastack/stack.go
commit f67e3cf9fd2ec0b35453ddadc1937c900f369503
Author: Victor Lowther <victor.lowther@gmail.com>
Date: Fri Aug 14 12:14:54 2020 -0500
test(parProfiles): Add staged transaction blocking tests.
Add a test to testParallelProfiles that tests transaction
unblocking at scale.
M backend/profiles_test.go
End of Note