Commit graph

202 commits

Author SHA1 Message Date
Bradley C. Kuszmaul
abd5212bcf Merge r26784 into the main line.
Changes include patches to
 * Handle gcc 4.4.1's foibles.
 * Instrument {{{toku_os_pread}}}
 * Release the lock during I/Os for queries (#3039)
 * Simplification of the ydb lock (no more backoff)
 * Use {{{DB_TXN_NOSYNC}} for several tests (which speeds them up).
 * Set the redzone to zero for several tests  (so they can run on machines with little free space). (#3111)
 * Add  helgrind and valgrind suppressions to some tests.
 * Add dollar-id-dollar to several files.
 * Declare the schedule status values to be volatile.
 * Add a method to the toku {{{RWLOCK}}} to effectively grab the read lock and then release it quickly.  (With only one condition variable acquisition.)
Fixes #3039. close[t:3039]
Refs #3111,   #3112,   #3119,   #3125,   #3126,   #3128,   #3131,   #3133,   #3142,   #3152.
   [t:3111] [t:3112] [t:3119] [t:3125] [t:3126] [t:3128] [t:3131] [t:3133] [t:3142] [t:3152].


git-svn-id: file:///svn/toku/tokudb@26785 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:34 -04:00
Barry Perlman
deb0d23050 [t:2949] Improve cachetable's get_and_pin_footprint.
git-svn-id: file:///svn/toku/tokudb@25399 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:26 -04:00
Barry Perlman
548d03d70e [t:2892] Merge upgrade logic to main. Merge command was svn merge --accept=postpone -r25293:HEAD ../tokudb.main+2892 .
git-svn-id: file:///svn/toku/tokudb@25303 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:25 -04:00
Rich Prohaska
24d915f259 #2966 use assert_zero in some of the newbrt files refs[t:2966]
git-svn-id: file:///svn/toku/tokudb@24717 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:23 -04:00
Rich Prohaska
5c0dce7cbb #2513 merge the cached threadpool to main refs[t:2513]
git-svn-id: file:///svn/toku/tokudb@24627 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:23 -04:00
Barry Perlman
db868fb31d [t:2947] Merge from tokudb.2947. Added errno to error log, don't take minicron lock for engine status. I want to print engine status to error log, but the env isn't available everywhere, so that's not here (yet).
git-svn-id: file:///svn/toku/tokudb@24075 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:23 -04:00
Bradley C. Kuszmaul
06d1c0e1eb Get rid of {{{TAGMALLOC}}}. Fixes #2912. close[t:2912]
git-svn-id: file:///svn/toku/tokudb@23493 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:22 -04:00
Rich Prohaska
a71c846cb4 #2877 refs[t:2877] remove kruft from the cachetable
git-svn-id: file:///svn/toku/tokudb@23180 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:22 -04:00
Yoni Fogel
0b0c76392a closes[t:1398] [t:2861] Remove more heaviside code (include straddle hacks) from newbrt
git-svn-id: file:///svn/toku/tokudb@23060 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:22 -04:00
Bradley C. Kuszmaul
5fcfafc691 Overactive assertion. Refs #2759. [t:2759]
git-svn-id: file:///svn/toku/tokudb@21573 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:20 -04:00
Bradley C. Kuszmaul
c49d7a8c0c The first loader gets 1/2 of cachetable memory.
* That's 2/3 of 3/4, which is calculated as 1/4 is unreservable.  The loader asks for 2/3 of what's reservable, getting one half.
The second loader gets 1/6 of cachetable memory.
 * That's 2/3 of 1/4: The loader asks for 2/3 of what's reservable.  1/2 has been reserved.  1/4 is unreservable.  1/4 is reservable.  2/3 of 1/4 is 1/6.
The third loader gets 1/18 of cachetable memory. 
 * That's 2/3 or 1/12.   (1/4 is unreservable. 1/2 was reserved by the first loader.  1/6 by the second.  That leaves 1/12)
Generally the next one gets 1/3 of what the previous one got.

Refs #2759. [t:2759]


git-svn-id: file:///svn/toku/tokudb@21564 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:19 -04:00
Bradley C. Kuszmaul
f635f4d458 Merge the fix for #2755 and #2759 to main. [t:2755] [t:2759]
{{{
svn merge -r21499:21502 https://svn.tokutek.com/tokudb/toku/tokudb.2755
}}}
.


git-svn-id: file:///svn/toku/tokudb@21503 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:19 -04:00
Bradley C. Kuszmaul
fa229ab979 Fix the cachetable-race so that it doesn't run in BDB and compiles under windows. Fixes #2699, #2698. close[t:2699] close[t:2698].
git-svn-id: file:///svn/toku/tokudb@20902 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:18 -04:00
Bradley C. Kuszmaul
aca0557856 Tell helgrind not to complain about the toku_checkpointing_user_status. Refs #2684. [t:2684]
git-svn-id: file:///svn/toku/tokudb@20879 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:18 -04:00
Bradley C. Kuszmaul
e43af77fc6 Create a test to trigger #2684 (the race found in checkpointing in loader-stress-test). [t:2684]
git-svn-id: file:///svn/toku/tokudb@20877 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:18 -04:00
Yoni Fogel
c9b2b066dd [t:2561] Merge tokudb.2561b to main. Disabled auto-upgrade and auto-upgrade tests
git-svn-id: file:///svn/toku/tokudb@20778 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:17 -04:00
Bradley C. Kuszmaul
c20a422a93 Reserve memory. Refs #2613. [t:2613].
git-svn-id: file:///svn/toku/tokudb@20241 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:12 -04:00
Barry Perlman
ae050bf956 [t:2504] Improve engine status.
git-svn-id: file:///svn/toku/tokudb@20077 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:11 -04:00
Bradley C. Kuszmaul
28cc5d1ed9 Merge the changes from 2499d onto the main line. Fixes #2499. close[t:2499].
{{{
svn merge -r 19523:19895 https://svn.tokutek.com/tokudb/toku/tokudb.2499d
}}}
.


git-svn-id: file:///svn/toku/tokudb@19902 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:09 -04:00
Yoni Fogel
8c34dda452 closes[t:2440] Merge #2440 to main.
closes[t:2512] merge #2512 to main
Delete dev branch

git-svn-id: file:///svn/toku/tokudb@19439 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:06 -04:00
Rich Prohaska
60d43a6d82 fix cachetable unpin and remove refs[t:2507]
git-svn-id: file:///svn/toku/tokudb@19264 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:06 -04:00
Yoni Fogel
fffb7ba30a [t:2506] [t:2507] Fix metadata (size writing) during write_pair_for_checkpoint
Also initialize the work item in the pair even if we actually are writing it.
Fix crash in getting size of empty FILENUMS

git-svn-id: file:///svn/toku/tokudb@19200 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:06 -04:00
Yoni Fogel
43aea2f126 [t:2449] do lru_touch on prefetch
git-svn-id: file:///svn/toku/tokudb@19195 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:06 -04:00
Yoni Fogel
2f1711d2a6 [t:2506] [t:2507] Forgot to initialize workitem for write_pair_for_checkpoint when enqueueing on cq
git-svn-id: file:///svn/toku/tokudb@19194 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:06 -04:00
Yoni Fogel
8bd25c46ab [t:2506] [t:2507] Fix locking mechanism for unpin_and_remove
We mistakenly thought that the workqueue lock was the same as the cachetable lock

git-svn-id: file:///svn/toku/tokudb@19188 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:06 -04:00
Yoni Fogel
3550e7dd38 [t:2506] [t:2507] Fix bug in toku_cachetable_unpin_and_remove
write_pair_for_checkpoint now uses completion queue if set

git-svn-id: file:///svn/toku/tokudb@19187 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:06 -04:00
Yoni Fogel
a5a29dbda6 [t:2506] [t:2507] Fix bug in toku_cachetable_unpin_and_remove
Correctly handles a blocked writer (checkpoint).  it checked state before state was set,
so incorrectly thought that there was never a blocked writer

git-svn-id: file:///svn/toku/tokudb@19186 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:06 -04:00
Yoni Fogel
e521486e82 [t:2506] [t:2507] Added some temporary asserts
git-svn-id: file:///svn/toku/tokudb@19182 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:06 -04:00
Yoni Fogel
1bf7a7a403 closes[t:2449] [t:2484] Merge #2449 changes to main.
Rollback logs are now checkpointed.  There are no rolltmp files.


git-svn-id: file:///svn/toku/tokudb@19167 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:05 -04:00
Barry Perlman
13c324ed06 Merge from 3.1.0 using following commands:
> svn merge --accept=postpone -r18405:HEAD https://svn.tokutek.com/tokudb/mysql.branches/3.0.5/tokudb
> svn merge --accept=postpone -r18484:HEAD https://svn.tokutek.com/tokudb/mysql.branches/3.1.0/tokudb
> svn merge --accept=postpone -c18403 https://svn.tokutek.com/tokudb/mysql.branches/3.0.5/tokudb
> all executed inside directory tokudb/toku/tokudb

git-svn-id: file:///svn/toku/tokudb@18877 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:04 -04:00
Yoni Fogel
b3f5306b41 [t:2249] Merge #2249 to main.
'load' entry and rollback/recovery,
single filename for brtloader and rest of brt
test-fsync

git-svn-id: file:///svn/toku/tokudb@18617 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:03 -04:00
Yoni Fogel
c6510b33fe closes[t:2463] Merge #2463 changes (r18552,18555) to main to fix race condition
with CACHEFILE->fd

git-svn-id: file:///svn/toku/tokudb@18612 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:02 -04:00
Dave Wells
0009757a37 merge changes to cachetable.c from tokudb.2445 [t:2444]
git-svn-id: file:///svn/toku/tokudb@18399 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:02 -04:00
Bradley C. Kuszmaul
178820a688 Merge changes from 2216a to main (except possibly for the windows subdirectory, which Yoni will do separately).
2216a is no longer valid.
Refs #2216. [t:2216].
{{{
svn merge -r17301:18179 https://svn.tokutek.com/tokudb/toku/tokudb.2216a/src/tests
}}}



git-svn-id: file:///svn/toku/tokudb@18181 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:01 -04:00
Rich Prohaska
1c7b2bce90 merge parallel block compressor from 3.0.4 to main closes[t:2371]
git-svn-id: file:///svn/toku/tokudb@18038 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:00 -04:00
Rich Prohaska
dd4d1192ff add checkpoint work in the cachetable to the wait time in get and pin. merge from 3.0.4 closes[t:2348]
git-svn-id: file:///svn/toku/tokudb@17819 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:59:00 -04:00
Bradley C. Kuszmaul
56d2c6fdf8 Merge 2216 onto main (from 16706-17293) as
{{{
svn merge -r16706:17293 https://svn.tokutek.com/tokudb/toku/tokudb.2216
}}}

Refs #2216. [t:2216]


git-svn-id: file:///svn/toku/tokudb@17299 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:59 -04:00
Yoni Fogel
838e0e5b1c [t:2318] Merge #2318 back to main up to r17166
git-svn-id: file:///svn/toku/tokudb@17178 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:59 -04:00
Bradley C. Kuszmaul
f27cc8fa09 Fixes #2323 by getting rid of gcc warnings. closes[t:2323]
git-svn-id: file:///svn/toku/tokudb@17147 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:59 -04:00
Barry Perlman
887a53259d Addresses #2090 refs[t:2090] Bugfix in cachetable_reserve_filenum(), added test program to exercise filenum reservation/release.
git-svn-id: file:///svn/toku/tokudb@15963 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:54 -04:00
Rich Prohaska
e8a671d611 imp toku_file_fsync in the port layer closes[t:1748]
git-svn-id: file:///svn/toku/tokudb@15805 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:52 -04:00
Yoni Fogel
357ebc950d Addresses #2172 refs[t:2172] Bugfix of commit r15040 (Addresses #2037 refs[t:2037])
Was not holding cachetable lock to access rwlock (whose mutex was cachetable lock).
This caused a race condition that you would sometimes grab the cachetable lock on exiting the lock but didn't have it before.

We believe this caused the deadlock

git-svn-id: file:///svn/toku/tokudb@15802 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:52 -04:00
Yoni Fogel
18c2c68f7c Addresses #2164 refs[t:2164] Zombie locking contract now enforced in rename/remove
Checkpoint holds entire brt open (as zombie if necessary) instead of just holding open the cachefile.

git-svn-id: file:///svn/toku/tokudb.2037b@15740 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:06 -04:00
Yoni Fogel
a522ceda16 Closes #2153 refs[t:2153] fopen logged only once (unless closed and reopened). Removed logger from cachefile_close,
removed txnid from logging of fopen.

git-svn-id: file:///svn/toku/tokudb.2037b@15691 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:06 -04:00
Yoni Fogel
a293c6dac5 Addresses #2090 refs[t:2090] Bugfix in reserving filenums during recovery.
Use filenum provided.

git-svn-id: file:///svn/toku/tokudb.2037b@15646 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:06 -04:00
Yoni Fogel
030df27e4d Addresses #2090 refs[t:2090] Logging for file create is done before actually creating file on disk.
We reserve a filenum before logging, and then use the reserved filenum when we actually open the cachefile.

git-svn-id: file:///svn/toku/tokudb.2037b@15645 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:06 -04:00
Barry Perlman
97fa5e2467 Addresses #2075, #2094 refs[t:2075] refs[t:2094]
Merge 2.2.0 scheduling statistics to 3.0.0 (#2075).
Restore #2094 fix to 3.0.0 (cachefiles_lock).
Command line for merge (executed in tokudb.2037b, ../tokudb.2.2.0 is local sandbox image of 2.2.0 release): svn merge -r14900:HEAD ../tokudb.2.2.0

git-svn-id: file:///svn/toku/tokudb.2037b@15629 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:06 -04:00
Rich Prohaska
6f6974aae9 port fairsched from 2.2.0 to main (for linux only) refs[t:2083]
git-svn-id: file:///svn/toku/tokudb@15573 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:04 -04:00
Yoni Fogel
580f5caafb Addresses #2037 refs[t:2037] Merged #2037 back into main
git-svn-id: file:///svn/toku/tokudb@15562 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:04 -04:00
Barry Perlman
c8b8e42194 Addresses 1792 refs[t:1792] Add more info to show engine status
git-svn-id: file:///svn/toku/tokudb@14804 c7de825b-a66e-492c-adef-691d508d4ae1
2013-04-16 23:58:03 -04:00