Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compaction causes previously deleted value to reappear #184

Closed
cmumford opened this issue Sep 9, 2014 · 5 comments
Closed

Compaction causes previously deleted value to reappear #184

cmumford opened this issue Sep 9, 2014 · 5 comments
Assignees

Comments

@cmumford
Copy link
Contributor

cmumford commented Sep 9, 2014

Original issue 178 created by kevin.d.regan on 2013-06-13T01:12:41.000Z:

What steps will reproduce the problem?

  1. Compile and run the included test program.
  2. Note that not all value have been properly deleted.
  3. Recompile the test program with the CompactRange call commented out.
  4. Note that all values have been properly deleted.

What is the expected output? What do you see instead?

The expected output from the program is:

Creating first key range
Creating second key range
Deleting second key range
Compacting database
Counting number of keys left
Found 0 keys

The actual output is:

Creating first key range
Creating second key range
Deleting second key range
Compacting database
Counting number of keys left
Found 1222706 keys

What version of the product are you using? On what operating system?

1.9, 1.10
FreeBSD, Ubuntu 12.10

Please provide any additional information below.

include <iostream>

include <sstream>

include <cstdlib>

include <leveldb/db.h>

include <leveldb/write_batch.h>

namespace {

const std::string DB_PATH = "/tmp/my_test_db";
size_t NUM_KEYS = 1100000;

void
check_status(const leveldb::Status& status, const std::string& error_msg)
{
if (!status.ok()) {
std::cerr << "ERROR: " << error_msg << ": " <<
status.ToString() << std::endl;
std::exit(1);
}
}

std::string
create_key_1(size_t i)
{
std::ostringstream out;
out << "my_key_" << i;
return out.str();
}

std::string
create_key_2(size_t i)
{
return create_key_1(i) + "_xxx";
}

} // namespace

int
main (int argc, char** argv)
{
// open database
leveldb::DB* db;
leveldb::Options db_options;
db_options.create_if_missing = true;
leveldb::Status status = leveldb::DB::Open(db_options, DB_PATH, &db);
check_status(status, "Could not open database");

// create first key range
std::cout &lt;&lt; &quot;Creating first key range&quot; &lt;&lt; std::endl;
leveldb::WriteBatch batch;
for (size_t i = 0; i &lt; NUM_KEYS; i++) {
    batch.Put(create_key_1(i), &quot;value for range 1 key&quot;);
}
status = db-&gt;Write(leveldb::WriteOptions(), &amp;batch);
check_status(status, &quot;Could not create keys for first range&quot;);

// create second key range
std::cout &lt;&lt; &quot;Creating second key range&quot; &lt;&lt; std::endl;
batch.Clear();
for (size_t i = 0; i &lt; NUM_KEYS; i++) {
    batch.Put(create_key_2(i), &quot;value for range 2 key&quot;);
}
status = db-&gt;Write(leveldb::WriteOptions(), &amp;batch);
check_status(status, &quot;Could not create keys for second range&quot;);

// delete second key range
std::cout &lt;&lt; &quot;Deleting second key range&quot; &lt;&lt; std::endl;
batch.Clear();
for (size_t i = 0; i &lt; NUM_KEYS; i++) {
    batch.Delete(create_key_2(i));
}
status = db-&gt;Write(leveldb::WriteOptions(), &amp;batch);
check_status(status, &quot;Could not delete keys&quot;);

// compact database
std::cout &lt;&lt; &quot;Compacting database&quot; &lt;&lt; std::endl;
db-&gt;CompactRange(NULL, NULL);

// count the keys
std::cout &lt;&lt; &quot;Counting number of keys left&quot; &lt;&lt; std::endl;
leveldb::Iterator* iter = db-&gt;NewIterator(leveldb::ReadOptions());;
size_t num_keys = 0;
for (iter-&gt;SeekToFirst(); iter-&gt;Valid(); iter-&gt;Next()) {
    num_keys++;
}
delete iter;
std::cout &lt;&lt; &quot;Found &quot; &lt;&lt; num_keys &lt;&lt; &quot; keys&quot; &lt;&lt; std::endl;

// close database
delete db;

}

@cmumford
Copy link
Contributor Author

cmumford commented Sep 9, 2014

Comment #1 originally posted by kevin.d.regan on 2013-06-13T01:16:23.000Z:

This seems to be related to the size of the database. Reducing the number of keys to 1000000 causes the example to work correctly. Conversely, increasing the value size for each entries causes the required number of keys necessary to trigger the error to go down.

@cmumford cmumford self-assigned this Sep 9, 2014
@cmumford
Copy link
Contributor Author

cmumford commented Sep 9, 2014

Comment #2 originally posted by kevin.d.regan on 2013-06-13T01:18:54.000Z:

Also, the overlap of keys was necessary to trigger the error (if I move the range 1 keys to a non-overlapping prefix, the problem went away).

@cmumford
Copy link
Contributor Author

cmumford commented Sep 9, 2014

Comment #3 originally posted by kevin.d.regan on 2013-06-13T01:19:39.000Z:

Expected output should be:

Creating first key range
Creating second key range
Deleting second key range
Compacting database
Counting number of keys left
Found 1200000 keys

@cmumford
Copy link
Contributor Author

cmumford commented Sep 9, 2014

Comment #4 originally posted by sanjay@google.com on 2013-06-13T19:32:35.000Z:

I have reproduced the problem and have a fix in progress. A new release should be out soon.

@cmumford
Copy link
Contributor Author

cmumford commented Sep 9, 2014

Comment #5 originally posted by dgrogan@chromium.org on 2013-06-13T23:30:35.000Z:

Fixed in 1.11.0.

@cmumford cmumford closed this as completed Sep 9, 2014
maochongxin pushed a commit to maochongxin/leveldb that referenced this issue Jul 21, 2022
DevinLeamy pushed a commit to DevinLeamy/mesos that referenced this issue Feb 15, 2024
LevelDB up to version 1.11 contained a bug [1] that sometimes could make
previously deleted keys reappear after compaction.

We should upgrade our bundled LevelDB to a more recent version to
benefit from bug fixes in it. However, version 1.14 introduced a change
that made rolling back to a previous LevelDB version harder. See
AURORA-21661 for details. So for now we upgrade to the most recent
version that is fully backwards compatible.

[1] google/leveldb#184

Conflicts:
	support/mesos-tidy/entrypoint.sh
DevinLeamy pushed a commit to DevinLeamy/mesos that referenced this issue Mar 14, 2024
LevelDB up to version 1.11 contained a bug [1] that sometimes could make
previously deleted keys reappear after compaction.

We should upgrade our bundled LevelDB to a more recent version to
benefit from bug fixes in it. However, version 1.14 introduced a change
that made rolling back to a previous LevelDB version harder. See
AURORA-21661 for details. So for now we upgrade to the most recent
version that is fully backwards compatible.

[1] google/leveldb#184

Conflicts:
	support/mesos-tidy/entrypoint.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant