간단한 rocksDB 예제
이 사이트는 페이스 북의 rocksDB와 python을 이용해서 간단하게 데이터 베이스 시스템의 성능을 측정하는 방식이다.
만약 rocksDB를 처음 접한다면 한번쯤 보고 따라 해보는 것도 좋을 거 같다.
순서는 대략 다음과 같이 진행된다.
-
git clone rocksDB github
-
make && make install 을 통한 rocksDB 설치
-
그리고 python을 설치 한후, 간단한 데이터베이스 코딩을 한다.
rocksDB test process
facebook github를 보면 rocksDB의 모든 것을 알 수 있다.
아래는 기본적인 설치 과정을 따라하면 경험한 오류 들이다.
자 이제 기본적인 설치 과정을 보려면 facebook rocksDB install.md or RocksDB install
위의 두 링크는 같은 것이기 때문에 어느 것을 봐도된다.
그리고 내가 진행한 방식은 두가지이다.
하나는 release version, 다른 하나는 그냥 rocksDB 공식 github 자체를 이용했다.
다른 말로는 rocksDB 4.3 & 4.6version이라는 말이다.
만약 4.3의 release rocksDB를 원한면 facebook git을 참조하자.
아니라면, 아래처럼 하자.
Now I have downloaded on facebook git with git instruction, which is “git clone https://github.com/facebook/rocksdb.git”.
And I compile the rocksDB source with make.
results are :
일단 처음으로 rocksDB를 설치를 하기 위아여 다운을 받고 있다.
[hyunyoung.lee@localhost ~]$ git clone https://github.com/facebook/rocksdb.git
[hyunyoung.lee@localhost ~]$ ls
Desktop Downloads linux-4.5 Music Public Templates
Documents leveldb linux-4.5.tar.xz Pictures rocksdb Videos
[hyunyoung.lee@localhost ~]$ cd rocksdb/
[hyunyoung.lee@localhost rocksdb]$ ls
appveyor.yml hdfs ROCKSDB_LITE.md
arcanist_util HISTORY.md src.mk
AUTHORS include table
build_tools INSTALL.md third-party
CMakeLists.txt java thirdparty.inc
CONTRIBUTING.md LANGUAGE-BINDINGS.md tools
coverage LICENSE USERS.md
db Makefile util
DEFAULT_OPTIONS_HISTORY.md memtable utilities
doc PATENTS Vagrantfile
DUMP_FORMAT.md port WINDOWS_PORT.md
examples README.md
[hyunyoung.lee@localhost rocksdb]$ make
Makefile:99: Warning: Compiling in debug mode. Don't use the resulting binary in production
GEN util/build_version.cc
[hyunyoung.lee@localhost rocksdb]$ ls
appveyor.yml db_test2 optimistic_transaction_test
arcanist_util db_universal_compaction_test options_file_test
arena_test db_wal_test options_test
AUTHORS DEFAULT_OPTIONS_HISTORY.md options_util_test
auto_roll_logger_test deletefile_test PATENTS
autovector_test delete_scheduler_test perf_context_test
backupable_db_test doc plain_table_db_test
block_based_filter_block_test document_db_test port
block_hash_index_test DUMP_FORMAT.md prefix_test
block_test dynamic_bloom_test rate_limiter_test
bloom_test env_test README.md
build_tools event_logger_test redis_test
cache_bench examples reduce_levels_test
cache_test fault_injection_test rocksdb_dump
checkpoint_test file_indexer_test ROCKSDB_LITE.md
CMakeLists.txt filelock_test rocksdb_undump
coding_test filename_test skiplist_test
column_family_test file_reader_writer_test slice_transform_test
compact_files_test flush_job_test spatial_db_test
compaction_iterator_test full_filter_block_test src.mk
compaction_job_stats_test geodb_test sst_dump
compaction_job_test hdfs sst_dump_test
compaction_picker_test heap_test stringappend_test
compact_on_deletion_collector_test histogram_test table
comparator_db_test HISTORY.md table_properties_collector_test
CONTRIBUTING.md include table_reader_bench
corruption_test inlineskiplist_test table_test
coverage INSTALL.md third-party
crc32c_test iostats_context_test thirdparty.inc
c_test java thread_list_test
cuckoo_table_builder_test json_document_test thread_local_test
cuckoo_table_db_test LANGUAGE-BINDINGS.md tools
cuckoo_table_reader_test ldb transaction_test
db ldb_cmd_test ttl_test
db_bench librocksdb_debug.a USERS.md
db_block_cache_test LICENSE util
db_compaction_filter_test listener_test utilities
db_compaction_test log_test Vagrantfile
db_dynamic_level_test make_config.mk version_builder_test
dbformat_test Makefile version_edit_test
db_inplace_update_test manual_compaction_test version_set_test
db_iter_test memenv_test wal_manager_test
db_log_iter_test memory_test WINDOWS_PORT.md
db_properties_test memtable write_batch_test
db_repl_stress memtable_list_test write_batch_with_index_test
db_sanity_test memtablerep_bench write_callback_test
db_stress merge_helper_test write_controller_test
db_table_properties_test merger_test write_stress
db_tailing_iter_test merge_test
db_test mock_env_test
[hyunyoung.lee@localhost rocksdb]$ ./db_bench
Please install gflags to run rocksdb tools.
위에서 본거와 같이 make을 하고 나면 각종 rocksDB와 관련되 실행 파일을 볼 수 있다.
그리고 벤치 마킹을 위해 ./db_bench프로그램을 실행했지만 rocksDB는 gflags에 dependency가 있다.
이를 설치해야한다.
for a few days, I was looking for the way, which installs gflags.
so I followed the install.md file of the facebook github.
in the gflag installation of the install.md file, their tools of rocksDB has denpendency on glfags library.
and I have stuided on the way to install gflags library.
First, direclty, I used the google github.
but google github source code need a camke(version 2.8.12) which creates makefile.
I installed cmake utility with yum. but this tools is 2.8.10.
So to fix it, I refers to this site, where here is to manually install cmake-3.0.0 on centos 7.
but, when I recompiled the rocksDB, I couldn’t run ./db_bench program.
that is why a executable file couldn’t open the .so file, ilbgflas.so.2.
if you have a question about the way to fix it, please refer to this.
The following is my experience for isntall.
아래를 보면 분명히 library는 존재하는데 찾을 수 없다는 메시지가 나온다.
- 이 경우는 올바른 라이브가 설치 되어도 나왔던메시지이다.
[hyunyoung.lee@localhost rocksdb]$ ./db_bench
./db_bench: error while loading shared libraries: libgflags.so.2: cannot open shared object file: No such file or directory
아래는 위의 에러를 수정한 후 다시 ./db_bench를 해봐도 같은 오류가 발생을 했다.
확인을 해보니 아래와 같은 파일을 있고, 정작 libgflags.so.2는 없다.
이를 확인 한 다음 나는 구글 github를 직접 이용을 한게 아니라
아예 페이스북에서 설명하는 방식을 하기로 했다.
[hyunyoung.lee@localhost build]$ cd /usr/
[hyunyoung.lee@localhost usr]$ ls
bin etc games include lib lib64 libexec local sbin share src tmp
[hyunyoung.lee@localhost usr]$ cd local/
[hyunyoung.lee@localhost local]$ ls
bin cmake-3.0.0 doc etc games include lib lib64 libexec sbin share src
[hyunyoung.lee@localhost local]$ cd lib
[hyunyoung.lee@localhost lib]$ ls
cmake libgflags_nothreads.so libgflags_nothreads.so.2.2 libgflags_nothreads.so.2.2.0 libgflags.so libgflags.so.2.2 libgflags.so.2.2.0 pkgconfig
아래는 페이스북이 알려준 gflags를 설치 library 위치를 확인한 결과이다.
[hyunyoung.lee@localhost ~]$ cd /usr/local/lib
[hyunyoung.lee@localhost lib]$ ls
libgflags.a libgflags_nothreads.a libgflags_nothreads.so libgflags_nothreads.so.2.1.0 libgflags.so.2 pkgconfig
libgflags.la libgflags_nothreads.la libgflags_nothreads.so.2 libgflags.so libgflags.so.2.1.0
[hyunyoung.lee@localhost lib]$
위의 에러를 고치기 위해서는 일단 리눅스의 동적라이브러리와 라이브러니 경로 설정을 제대로 해야 한다.
그 결과 정상적으로 설치가 가능했다. ./db_bench프로그램을 위한 정상적이 라이브러리 설치가 가능했다.
그리고 아래는 이제 제대로 ./db_bench프로그램으로 ssd를 테스트 해본것이다.
[hyunyoung.lee@localhost rocksdb]$ ./db_bench
LevelDB: version 4.6
Date: Wed Apr 6 19:46:33 2016
CPU: 4 * Intel(R) Core(TM) i3-0000 CPU @ 3.40GHz
CPUCache: 3072 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
Prefix: 0 bytes
Keys per prefix: 0
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
Write rate: 0 bytes/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 0
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
DB path: [/tmp/rocksdbtest-1000/dbbench]
fillseq : 1.716 micros/op 582671 ops/sec; 64.5 MB/s
DB path: [/tmp/rocksdbtest-1000/dbbench]
fillsync : 4.138 micros/op 241635 ops/sec; 26.7 MB/s (1000 ops)
DB path: [/tmp/rocksdbtest-1000/dbbench]
fillrandom : 4.358 micros/op 229486 ops/sec; 25.4 MB/s
DB path: [/tmp/rocksdbtest-1000/dbbench]
overwrite : 4.306 micros/op 232244 ops/sec; 25.7 MB/s
DB path: [/tmp/rocksdbtest-1000/dbbench]
readrandom : 5.271 micros/op 189712 ops/sec; 21.0 MB/s (1000000 of 1000000 found)
DB path: [/tmp/rocksdbtest-1000/dbbench]
newiterator : 0.614 micros/op 1628417 ops/sec;
DB path: [/tmp/rocksdbtest-1000/dbbench]
newiteratorwhilewriting : 0.677 micros/op 1476487 ops/sec;
DB path: [/tmp/rocksdbtest-1000/dbbench]
seekrandom : 10.139 micros/op 98630 ops/sec; (816350 of 1000000 found)
DB path: [/tmp/rocksdbtest-1000/dbbench]
seekrandomwhilewriting : 18.241 micros/op 54821 ops/sec; (795757 of 1000000 found)
DB path: [/tmp/rocksdbtest-1000/dbbench]
put or merge error: Not implemented: Provide a merge_operator when opening DB
pure virtual method called
pure virtual method called
terminate called without an active exception
terminate called recursively
Received signal 6 (Aborted)
Aborted
[hyunyoung.lee@localhost rocksdb]$ vim ./bashrc
[hyunyoung.lee@localhost rocksdb]$ gedit ~/.bashrc
[hyunyoung.lee@localhost rocksdb]$ ./db_bench
LevelDB: version 4.6
Date: Thu Apr 7 09:53:27 2016
CPU: 4 * Intel(R) Core(TM) i3-0000 CPU @ 3.40GHz
CPUCache: 3072 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
Prefix: 0 bytes
Keys per prefix: 0
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
Write rate: 0 bytes/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 0
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
DB path: [/tmp/rocksdbtest-1000/dbbench]
fillseq : 1.771 micros/op 564609 ops/sec; 62.5 MB/s
DB path: [/tmp/rocksdbtest-1000/dbbench]
fillsync : 3.994 micros/op 250388 ops/sec; 27.7 MB/s (1000 ops)
DB path: [/tmp/rocksdbtest-1000/dbbench]
fillrandom : 4.313 micros/op 231867 ops/sec; 25.7 MB/s
DB path: [/tmp/rocksdbtest-1000/dbbench]
overwrite : 4.379 micros/op 228341 ops/sec; 25.3 MB/s
DB path: [/tmp/rocksdbtest-1000/dbbench]
readrandom : 5.324 micros/op 187812 ops/sec; 20.8 MB/s (1000000 of 1000000 found)
DB path: [/tmp/rocksdbtest-1000/dbbench]
newiterator : 0.607 micros/op 1648277 ops/sec;
DB path: [/tmp/rocksdbtest-1000/dbbench]
newiteratorwhilewriting : 0.653 micros/op 1531684 ops/sec;
DB path: [/tmp/rocksdbtest-1000/dbbench]
seekrandom : 10.463 micros/op 95577 ops/sec; (816350 of 1000000 found)
DB path: [/tmp/rocksdbtest-1000/dbbench]
seekrandomwhilewriting : 19.673 micros/op 50829 ops/sec; (795962 of 1000000 found)
DB path: [/tmp/rocksdbtest-1000/dbbench]
put or merge error: Not implemented: Provide a merge_operator when opening DB
pure virtual method called
pure virtual method called
terminate called without an active exception
terminate called recursively
Received signal 6 (Aborted)
Aborted
[hyunyoung.lee@localhost rocksdb]$