If you finish installing the QEMU-NVMe, Now You can use the newest version virtual OpenChannelSSD.
i.e You can try to use LightNVM with virtual OpenChannelSSD.
Befoer starting this process, I recommend you to read Getting started of the official lightnvm site.
As you can see the official lightnvm site.
you can know the following :
-
What version of linux support lightnvm ?
-
What tool do you need to install LightNVM on QEMU-NMVe. ?
-
Of course. How Can I install ane execute QEMU-NVMe for OpenChannelSSD ?
First, I realized What I misunderstan about virt-manager.
I think, with GUI,virt-manager, I can use QEMU-NVMe with virt-Manager. But I realized that is wrong.
from Redhat site written in 20 Apr 2016 on redHat board.
Becuase, When you use virtualization of kernel, you use Libvirt(the virtualization API).
So Libvirt of Redhat on CentOS 7 doesn’t even support it explicitly yet. So I can understan I have to use command Line to execute NVMe device. i.e virt-manager doesn’t support how to add NVMe device.
So I will use QEMU-NVMe for openchannelSSD on command line.
Before starting installing lighiNVM, so far, I will brief what I have done about QEMU-KVM and QEMU-NVMe.
Summary of installing QEMU-KVM on HOST OS
$ sudo yum install -y qemu-kvm qemu-img virt-manager libvirt libvirt-python libvirt-client virt-install (virt-viewer)
-- instead of the above command, I recommend you to contain virt-viewer. rather, you have to install something related to virt-viewer. So what I recommend is :
$ sudo yum install -y qemu-kvm qemu-img virt-manager libvirt libvirt-python libvirt-client virt-install virt-viewer
$ reboot OR shutdown -r now
$ virt-manager
The above command is basic setting to execute QEMU-KVM with GUI, virt-manager tool.
And if you want to how to install GUEST OS, please see my blog’s QEMU-URL
Of course, When you install “Development Tools” with yum to compile kernel.
If you don’t install “Development Tools”, please add one line first like this
$ sudo yum groupinstall “Development Tools”
Summary of install and compiling QEMU-NVMe for OpenChannelSSD on HOST OS
- first you need to install Qemu-nvme
- If you already installed "Development tools", just do it as follows.
$ git clone https://github.com/OpenChannelSSD/qemu-nvme.git
$ cd qemu-nvme
- this is my chocie when I evaluate FIO 4k ramdomwrit T1Q256 FOX(fresh of box)
$ ./configure --python=/usr/bin/python2 --enable-kvm --target-list=x86_64-softmmu --enable-linux-aio --prefix=$HOME/qemu-nvme
- At this time. You find error
- solution for that is below
$ sudo yum install -y gtk2-devel OR $ sudo yum install libaio-devel
- One more time, ./configure ~~~ . After that
$ make -j8
$ sudo make install
- if you want to use Ramdisk
$ mkdir ./ramdisk OR $ mkdir ./rootOfRamdisk
$ sudo mount -t tmpfs -o size=7G tmpfs ./ramdisk OR $ sudo mount -t tmpfs -o size=7G tmpfs ./rootOfRamdisk
$ cd ramdisk OR $ cd rootOFRamdisk
- for Qemu-nvme
$ dd if=/dev/zero of=./blknvme bs=1G count=4
$ ~/qemu-nvme/bin/qemu-system-x86_64 -m 16G -smp 4 --enable-kvm -hda /var/lib/libvirt/images/vDisk.qcow2(virtual device) -drive file=./blknvme(virtual nvme),if=none,id=mynvme -device nvme,drive=mynvme,serial=deadbeef,lver=1,~~~(another option) -monitor stdio
- For example command
$ ~/qemu-nvme/bin/qemu-system-x86_64 -m 16G -smp 4 --enable-kvm -hda /var/lib/libvirt/images/vDisk.qcow2 -drive file=./blknvme,if=none,id=mynvme -device nvme,drive=mynvme,serial=deadbeef,lver=1,ll2pmode=0,lnum_lun=4,lnum_pln=2,lsecs_per_pg=4,lpgs_per_blk=512 -monitor stdio
GuestOS’s status of QEMU-KVM before installing QEMU-NVMe.
On QEMU-KVM, you can perceive NVMe device and PCI related to NVMe.
But on QEMU-NVMe, the everything is perceived. it’s amazing,
The following is status of lspci, lsblk, uname, dmesg, ls /dev to check if there is NVMe device on QEMU-KVM
i.e the following data is measured on GUEST OS
Kernel Version
$ uname -a
Linux localhost.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)
dmesg | grep ‘nvm’
$ dmesg | grep 'nvm'
There is nothing to print
lspci -vt
$ lspci -vt
-[0000:00]-+-00.0 Intel Corporation 440FX - 82441FX PMC [Natoma]
+-01.0 Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
+-01.1 Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
+-01.3 Intel Corporation 82371AB/EB/MB PIIX4 ACPI
+-02.0 Red Hat, Inc. QXL paravirtual graphic card
+-03.0 Red Hat, Inc Virtio network device
+-04.0 Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) High Definition Audio Controller
+-05.0 Red Hat, Inc Virtio console
+-06.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
+-06.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
+-06.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
+-06.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
+-07.0 Red Hat, Inc Virtio block device
\-08.0 Red Hat, Inc Virtio memory balloon
lsblk
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 32G 0 disk
├─vda1 252:1 0 500M 0 part /boot
└─vda2 252:2 0 31.5G 0 part
├─centos-root 253:0 0 28.3G 0 lvm /
└─centos-swap 253:1 0 3.2G 0 lvm [SWAP]
ls /dev
$ ls /dev
autofs fd mqueue snapshot tty19 tty35 tty51 ttyS1 vcsa
block full net snd tty2 tty36 tty52 ttyS2 vcsa1
bsg fuse network_latency sr0 tty20 tty37 tty53 ttyS3 vcsa2
btrfs-control hidraw0 network_throughput stderr tty21 tty38 tty54 uhid vcsa3
bus hpet null stdin tty22 tty39 tty55 uinput vcsa4
cdrom hugepages nvram stdout tty23 tty4 tty56 urandom vcsa5
centos initctl oldmem tty tty24 tty40 tty57 usbmon0 vcsa6
char input port tty0 tty25 tty41 tty58 usbmon1 vda
console kmsg ppp tty1 tty26 tty42 tty59 usbmon2 vda1
core log ptmx tty10 tty27 tty43 tty6 usbmon3 vda2
cpu loop-control pts tty11 tty28 tty44 tty60 usbmon4 vfio
cpu_dma_latency lp0 random tty12 tty29 tty45 tty61 vcs vga_arbiter
crash lp1 raw tty13 tty3 tty46 tty62 vcs1 vhost-net
disk lp2 rfkill tty14 tty30 tty47 tty63 vcs2 virtio-ports
dm-0 lp3 rtc tty15 tty31 tty48 tty7 vcs3 vport1p1
dm-1 mapper rtc0 tty16 tty32 tty49 tty8 vcs4 vport1p2
dri mcelog sg0 tty17 tty33 tty5 tty9 vcs5 zero
fb0 mem shm tty18 tty34 tty50 ttyS0 vcs6
How to insatll and set up LightNVM on QEMU-NVMe for OpenChannelSSD on GUETOS
AS you can see getting started of Lightnvm.io
the portion of install kernel 4.4+ in getting started of LightNVM.
You can find “LightNVM is directly supported in Linux kernel 4.4+.”
So you need kernel above 4.4+ version. Fortunately, LightNVM guys manage kernel source for OpenChannleNVMe and LightNVM
Also You need lnvm tool. as you can see the above figure.
As you can see the above image. You know How to run lnvm tool. Aslo You can check how to use lnvm tool in LightNVM site’s Command Line
To sum up, lnvm tool connect among NVMe driver, generic media driver and target.
i.e you can make your own FTL like pblk and your own generic driver to fit your device(SSD).
After that, Through lnvm, You connect each other(target and media manager).
Before using the lnvm tool, you have to get sudo privilege.
From now on, let’s install and kernel complie for LightNVM and OpenChannelSSD
The newest Kernel compile on GUESTOS of QEMU-NVMe.
Also LightNVM guys manage Linux kernel support of OpenChannelSSD.
first git clone and kernel build on GuestOS
$ git clone https://github.com/OpenChannelSSD/linux.git
cd linux
cp /.config(The Original location) /.config(OpenChannelSSD’s Linux)
make olddefconfig
make -j8
sudo make modules_install
sudo make install
be careful when you choose options in menuconfig
the following is based on rrpc but currently I evaluate on pblk.
CONFIG_NVM=y
CONFIG_NVM_DEBUG=Y
CONFIG_NVM_GENNVM=y
CONFIG_NVM_RRPC=y
The above four things is the same.
BUT, I divided into two cases.
One is :
CONFIG_BLK_DEV_NULL_BLK=m
CONFIG_BLK_DEV_NVME=m
The other is :
CONFIG_BLK_DEV_NULL_BLK=m
CONFIG_BLK_DEV_NVME=y
That doesn’t matter When it comes to creating lightNVM targets and device.
If you detect a problem. I recommend you that You will install once again after erase the several file
Those files is :
bbtable.qemu & blkbme(virtual device) & meta.qemu
but if you consider symbol dependency, when you build kernel.
I will explain you based on pblk
CONFIG_NVM=y
# Expose the /sys/module/lnvm/parameters/configure_debug interface
CONFIG_NVM_DEBUG=y
# Target support (required to expose the open-channel SSD as a block device)
CONFIG_NVM_PBLK=y
# Do not perform recovery upon boot (Disables recovery of the mapping table upon boot. Disable when using across boots)
CONFIG_NVM_PBLK_NO_RECOV=y
# generic media manager support (required)
CONFIG_NVM_GENNVM=y
# For NVMe support
CONFIG_BLK_DEV_NVME=y
as you can see the above.
# For NVMe support CONFIG_BLK_DEV_NVME=y has dependency of symbol.
So I recommend you
in make menuconfig, Device Drivers-> NVM EXPRESS block device should be [*].
maybe if you want to check up what is different when it is done or not.
I recommend you that
you can understand olddefconfig meaning and symbol dependency.
Second usage of lnvm on GuestOS
$ git clone https://github.com/OpenChannelSSD/lnvm.git
$ git clone https://github.com/OpenChannelSSD/lnvm.git
Cloning into 'lnvm'...
remote: Counting objects: 80, done.
remote: Total 80 (delta 0), reused 0(delta 0), pack-reused 80
$ cd lnvm/
from here, the whole thing is impelemented in lnvm directory
$ ls
debian Documentation LICENSE linux lnvm.c Makefile README.md
$ make
cc -m64 -std=gnu99 -02 -g -pthread -D_GNU_SOURCE -Wall lnvm.c -lm -o lnvm
$ usdo make install
install -d /usr/local/sbin
install -m 755 lnvm /usr/local/sbin
When using lnvm tool, error can happen like this
$ lnvm devices
Failed to open LightNVM mgmt /dev/lightnvm/control. Error: -1
At that time, just change command like this
$ sudo ./lnvm devices
Number of devices: 1
Device Block manager version
nvme0n1 none (0,0,0)
the following is the same error and solution.
$ lnvm info
Failed to open LightNVM mgmt /dev/lightnvm/control. Error: -1
$ ./lnvm info or $ ./lnvm -d init -d nvme0n1
Failed to open LightNVM mgmt /dev/lightnvm/control. Error: -1
$ sudo ./lnvm info
LightNVM (1,0,0). 2 target type(s) regitstered
Type Version
pblk (1,0,0)
rrpc (1,0,0)
$ lnvm init -d nvme0n1
Failed to open LightNVM mgmt /dev/lightnvm/control. Error: -1
The followings is the final command to connect among NVMe driver, media manager and target
$ sudo ./lnvm init -d nvme0n1
$ sudo ./lnvm create -d nvme0n1 -n hyun -t rrpc (-o 0:3)
-o means the number of lnus.
if you type lsblk on command line, you can check hyun device generated on blknvme with dd command.
Be careful, when you make target and boot with QEMU-NVMe, you have to do that in directory contain blknvme that is virtual NVMe device.
So, meta.qemu and badblock table is created. when turning on kernel with QEMU-NVMe and creating target like this.
currently, you can see meta.qemu
blknvme and metat.qemu file appears.
$ ls
bbtable.qemu blknvme meta.qemu
Summary of installing lightNVM and compiling kernel on GUESTOS of QEMU-NVMe
On GEUST OS (Ctrl+Alt+G)
- You have to install linux for openchannelSSD
$ git clone https://github.com/OpenChannelSSD/linux.git
- After the above, You have to install kernel.
- cp /.config(The Original location) /.config(OpenChannelSSD's Linux)
- make oldconfig or olddefconfig
- make menucofig -> configuration for LihgtNVM pblk
- make -j8
- sudo make modules_install
- sudo make modules
- for lnvm
$ git clone https://github.com/OpenChannelSSD/lnvm.git
$ cd lnvm
$ make
$ sudo make install
- After completing installation of lnvm, for looking at usage
- in lnvm directory
$ sudo ./lnvm --help
$ sudo ./lnvm info
$ sudo ./lnvm devices
$ sudo ./lnvm init -d nvme0n1
$ sudo ./lnvm create -d nvme0n1 -n hyun(target name) -t pblk -o 0:3
$ sudo ./lnvm remove -n hyun(targer name)
Summary of the whole process to use lightNVM on QEMU-NVMe
as of 20161205.
On host side
- If you don't install qemu-kvm, first, You have to install that tool
$ sudo yum install -y qemu-kvm qemu-img virt-manager libvirt libvirt-python libvirt-client virt-install (virt-viewer)
$ reboot
$ virt-manager
- first you need to install Qemu-nvme
- If you already installed "Development tools", just do it as follows.
$ git clone https://github.com/OpenChannelSSD/qemu-nvme.git
$ cd qemu-nvme
$ ./configure --python=/usr/bin/python2 --enable-kvm --target-list=x86_64-softmmu --enable-linux-aio --prefix=$HOME/qemu-nvme
- At this time. You find error
- solution for that is below
$ sudo yum install -y gtk2-devel OR $ sudo yum install libaio-devel
- One more time, ./configure ~~~ . After that
$ make -j8
$ sudo make install
- if you want to use Ramdisk
$ mkdir ./ramdisk OR $ mkdir ./rootOfRamdisk
$ sudo mount -t tmpfs -o size=7G tmpfs ./ramdisk OR $ sudo mount -t tmpfs -o size=7G tmpfs ./rootOfRamdisk
$ cd ramdisk OR $ cd rootOFRamdisk
- for Qemu-nvme
$ dd if=/dev/zero of=./blknvme bs=1G count=4
$ sudo ~/qemu-nvme/bin/qemu-system-x86_64 -m 16G -smp 4 --enable-kvm -hda /var/lib/libvirt/images/vDisk.qcow2(virtual device) -drive file=./blknvme(virtual nvme),if=none,id=mynvme -device nvme,drive=mynvme,serial=deadbeef,lver=1,~~~(another option) -monitor stdio
- For example command
$ sudo ~/qemu-nvme/bin/qemu-system-x86_64 -m 16G -smp 4 --enable-kvm -hda /var/lib/libvirt/images/vDisk.qcow2 -drive file=./blknvme,if=none,id=mynvme -device nvme,drive=mynvme,serial=deadbeef,lver=1,ll2pmode=0,lnum_lun=4,lnum_pln=2,lsecs_per_pg=4,lpgs_per_blk=512 -monitor stdio
On GeustOS(Ctrl+Alt+G)
- You have to install linux for openchannelSSD
$ git clone https://github.com/OpenChannelSSD/linux.git
- After the above, You have to install kernel.
- cp /.config(The Original location) /.config(OpenChannelSSD's Linux)
- make oldconfig or olddefconfig
- make menucofig -> configuration for LihgtNVM pblk - you always have to check getting started of lighinvm to know what option is to use LightNVM and OpenChannelSSD.
- make -j8
- sudo make modules_install
- sudo make modules
- for lnvm
$ git clone https://github.com/OpenChannelSSD/lnvm.git
$ cd lnvm
$ make
$ sudo make install
- After completing installation of lnvm, for looking at usage
$ sudo ./lnvm --help
$ sudo ./lnvm info
$ sudo ./lnvm devices
$ sudo ./lnvm init -d nvme0n1
$ sudo ./lnvm create -d nvme0n1 -n hyun(target name) -t pblk -o 0:3
$ sudo ./lnvm remove -n hyun(targer name)
- Simple test with Fio
- Installation of fio
- fio options in shell script
BLOCKSIZE=4k
RW=randwrite
for REPEAT in $(seq 0 9)
do
fio --name=${BLOCKSIZE}-${RW}-${REPEAT} \
--filename=/dev/nvme0n1 \
--rw=randwrite \
--bs=4096 \
--ioengine=libaio \
--numjobs=1 \
--iodepth=256 \
--direct=1 \
--thread \
--write_iops_log=${BLOCKSIZE}-${RW}-${REPEAT} \
--log_avg_msec=1000 \
--size=$((900*1024*1204)) or --size=50% or Noone
done
Reference
-
lightnvm.io - the official site of lightnvm
The following is explaining How to add NVMe device by virt-manager, the answer is you cannot use virt-manger.
Source code for openchannelSSD
Appendix A : build Kernel to latest version
AS of 20161123 below image is changed for pblk on officail site of lightNVM
In order to work on lightnvm, I need kernel above version 4.4+.**
I will install kernel version latest version
you remeber the above image is change to fit pblk on the officail site of lightNVM
In my build process
su
chmod u+w /etc/sudoers
vim /etc/sudoers
chmod u-w /etc/sudoers
sudo yum install -y gcc bc openssl-devel ncurses-devel pciutils libudev-devel
wget https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.8.tar.xz
xz -cd linux-4.X.tar.xz | tar xvf -
$ cp ./Downloads/config-4.8.0-1.el7.elrepo.x86_64 ./linux-4.8/.config
$ cd linux-4.8/
(make oldconfig)
$ make menuconfig
$ make
$ make modules_install install
($ grep ^menuentry /boot/grub2/grub.cfg | cut –d\’ –f2)
$ grub2-editenv list
$ grub2-set-default 0
$ sudo grep ^menuentry /boot/grub2/grub.cfg | cut –d\’ –f2
cut: you must specify a list of bytes, characters, or fields
Try 'cut --help' for more information.
[sudo] password for hyunyoung.lee:
-- configure procedure of booting kernel.
$ sudo grub2-editenv list
saved_entry=CentOS Linux (3.10.0-327.el7.x86_64) 7 (Core)
$ sudo grub2-set-default 0
kernel build to latest version with OpenchannelSSD
Appendix B
The following is build mainline kerenl of ELRepo on HostOS
$ sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
$ sudo rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
Retrieving http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
Preparing... ################################# [100%]
Updating / installing...
1:elrepo-release-7.0-2.el7.elrepo ################################# [100%]
$ sudo yum --enablerepo=elrepo-kernel install kernel-ml
Loaded plugins: fastestmirror, langpacks
elrepo | 2.9 kB 00:00
elrepo-kernel | 2.9 kB 00:00
(1/2): elrepo/primary_db | 376 kB 00:00
(2/2): elrepo-kernel/primary_db | 1.6 MB 00:00
Loading mirror speeds from cached hostfile
* base: centos-distro.1gservers.com
* elrepo: elrepo.org
* elrepo-kernel: elrepo.org
* epel: linux.mirrors.es.net
* extras: mirrors.usc.edu
* updates: centos.sonn.com
Resolving Dependencies
--> Running transaction check
---> Package kernel-ml.x86_64 0:4.8.1-1.el7.elrepo will be installed
--> Finished Dependency Resolution
......
$ sudo yum --enablerepo=elrepo-kernel update
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: centos-distro.1gservers.com
* elrepo: repos.lax-noc.com
* elrepo-kernel: repos.lax-noc.com
* epel: mirror.prgmr.com
* extras: mirrors.usc.edu
* updates: centos.sonn.com
Resolving Dependencies
--> Running transaction check
---> Package python-perf.x86_64 0:3.10.0-327.36.2.el7 will be updated
---> Package python-perf.x86_64 0:4.8.1-1.el7.elrepo will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Updating:
python-perf x86_64 4.8.1-1.el7.elrepo elrepo-kernel 443 k
Transaction Summary
================================================================================
......