NVMe-Cli
Now, I’m looking at commit version,“0142aebd1125ae33cdf60fa957a8bfef5851a76e”, on master brach,
This tool made of OpenSource is for NVMe device.
You can do anything of command of NVMexpression specification 1.2 and so on.
it’s like this
$ sudo ./nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 SN IS ~~~~~~~~ Model's name of SSD 1 1000.00 GB / 1000.00 GB 512 B + 0 B Ver of FW
OR
$ sudo ./nvme smart-log /dev/nvme0
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning : 0
temperature : 00 C
available_spare : 000%
available_spare_threshold : 00%
percentage_used : 0%
data_units_read : 00
data_units_written : 00,000,000
host_read_commands : 000
host_write_commands : 0,000,000,000
controller_busy_time : 000
power_cycles : 0
power_on_hours : 000
unsafe_shutdowns : 0
media_errors : 0
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 0 C
Temperature Sensor 2 : 0 C
Temperature Sensor 3 : 0 C
Temperature Sensor 4 : 0 C
Temperature Sensor 5 : 0 C
Temperature Sensor 6 : 0 C
Temperature Sensor 7 : 0 C
Temperature Sensor 8 : 0 C
As you can see the above command of nvme-cli, you can get information of nvme device with NVMe-cli tool.
So far, The above nvme-cli from the commit version is similar to NVMe-cli ver1.1 release.
Later on, I would make VU command with NVMe-cli ver 1.1 opensource based on experience with NVMe-cli ver 0.7
In other words, according to NVMe specification, The common command of all NVMe device is included in NVMe-cil tool.
Also, You can tune the NVMe-cli of opensource to make the fit VU command in addition.
let’s see NVME-Cli ver 1.1 release
NVMe-Cli ver 1.1 release
NVMe-Cli-ver 0.7
$ cd nvme-cli--0.7
$ ls -v
CONTRIBUTING.md NVME-VERSION-GEN linux nvme-ioctl.c regress
Documentation README.md nvme.c nvme-ioctl.h scripts
LICENSE completions nvme.control.in nvme-print.c src
Makefile debian nvme.spec.in nvme-print.h
$ cd linux
$ ls -v
nvme.h
and
NVMe-Cli-ver 1.1
$ cd nvme-cli-1.1
$ ls -v
CONTRIBUTING.md NVME-VERSION-GEN cmd.h debian intel-nvme.c linux memblaze-nvme.h nvme.spec.in nvme-lightnvm.c nvme-print.c plugin.c suffix.c
Documentation README.md cmd_handler.h define_cmd.h intel-nvme.h lnvm-nvme.c nvme.c nvme-builtin.h nvme-lightnvm.h nvme-print.h plugin.h suffix.h
LICENSE argconfig.c common.h fabrics.c json.c lnvm-nvme.h nvme.control.in nvme-ioctl.c nvme-models.c parser.c regress tests
Makefile argconfig.h completions fabrics.h json.h memblaze-nvme.c nvme.h nvme-ioctl.h nvme-models.h parser.h scripts
$ cd linux
$ ls -v
lightnvm.h nvme.h nvme_ioctl.h
As you can compare NVMe-cli ver 1.1 with ver 0.7, many things is changed in NVMe-cli tool.
I think because of change of mainline of kernel.
Now, The mainline kernel supports NVMe fabric and LightNVM(Open-ChannelSSD).
So owing to NVMe Fabric, NVMe-cli tool supports json.c and fabrics.c and so on.
And you can see lnvm-nvme.c and lightnvm.h of linux directory as well.
You can notice the trend of NVMe device changes to lightNVM and Fabrics.
Also, as you can see lnvm-nvme.c file. I think the big trend of NVMe is Open-ChannelSSD and lightNVM.
./nvme between ver 1.1 and ver 0.7 of NVMe-cli
NVMe-Cli-ver 0.7
$ ./nvme --help
nvme-0.7
usage: nvme <command> [<device>] [<args>]
The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).
The following are all implemented sub-commands:
list List all NVMe devices and namespaces on machine
id-ctrl Send NVMe Identify Controller
id-ns Send NVMe Identify Namespace, display structure
list-ns Send NVMe Identify List, display structure
create-ns Creates a namespace with the provided parameters
delete-ns Deletes a namespace from the controller
attach-ns Attaches a namespace to requested controller(s)
detach-ns Detaches a namespace from requested controller(s)
list-ctrl Send NVMe Identify Controller List, display structure
get-ns-id Retrieve the namespace ID of opened block device
get-log Generic NVMe get log, returns log in raw format
fw-log Retrieve FW Log, show it
smart-log Retrieve SMART Log, show it
smart-log-add Retrieve additional SMART Log, show it
error-log Retrieve Error Log, show it
get-feature Get feature and show the resulting value
set-feature Set a feature and show the resulting value
format Format namespace with new block format
fw-activate Activate new firmware slot
fw-download Download new firmware
admin-passthru Submit arbitrary admin command, return results
io-passthru Submit an arbitrary IO command, return results
security-send Submit a Security Send command, return results
security-recv Submit a Security Receive command, return results
resv-acquire Submit a Reservation Acquire, return results
resv-register Submit a Reservation Register, return results
resv-release Submit a Reservation Release, return results
resv-report Submit a Reservation Report, return results
dsm Submit a Data Set Management command, return results
flush Submit a Flush command, return results
compare Submit a Compare command, return results
read Submit a read command, return results
write Submit a write command, return results
write-zeroes Submit a write zeroes command, return results
write-uncor Submit a write uncorrectable command, return results
reset Resets the controller
subsystem-reset Resets the controller
show-regs Shows the controller registers. Requires admin character device
version Shows the program version
help Display this help
See 'nvme help <command>' for more information on a specific command.
NVMe-Cli-ver 1.1
$ ./nvme --help
nvme-1.1
usage: nvme <command> [<device>] [<args>]
The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).
The following are all implemented sub-commands:
list List all NVMe devices and namespaces on machine
id-ctrl Send NVMe Identify Controller
id-ns Send NVMe Identify Namespace, display structure
list-ns Send NVMe Identify List, display structure
create-ns Creates a namespace with the provided parameters
delete-ns Deletes a namespace from the controller
attach-ns Attaches a namespace to requested controller(s)
detach-ns Detaches a namespace from requested controller(s)
list-ctrl Send NVMe Identify Controller List, display structure
get-ns-id Retrieve the namespace ID of opened block device
get-log Generic NVMe get log, returns log in raw format
fw-log Retrieve FW Log, show it
smart-log Retrieve SMART Log, show it
error-log Retrieve Error Log, show it
get-feature Get feature and show the resulting value
set-feature Set a feature and show the resulting value
format Format namespace with new block format
fw-activate Activate new firmware slot
fw-download Download new firmware
admin-passthru Submit arbitrary admin command, return results
io-passthru Submit an arbitrary IO command, return results
security-send Submit a Security Send command, return results
security-recv Submit a Security Receive command, return results
resv-acquire Submit a Reservation Acquire, return results
resv-register Submit a Reservation Register, return results
resv-release Submit a Reservation Release, return results
resv-report Submit a Reservation Report, return results
dsm Submit a Data Set Management command, return results
flush Submit a Flush command, return results
compare Submit a Compare command, return results
read Submit a read command, return results
write Submit a write command, return results
write-zeroes Submit a write zeroes command, return results
write-uncor Submit a write uncorrectable command, return results
reset Resets the controller
subsystem-reset Resets the controller
show-regs Shows the controller registers. Requires admin character device
discover Discover NVMeoF subsystems
connect-all Discover and Connect to NVMeoF subsystems
connect Connect to NVMeoF subsystem
disconnect Disconnect from NVMeoF subsystem
version Shows the program version
help Display this help
See 'nvme help <command>' for more information on a specific command
The following are all installed plugin extensions:
intel Intel vendor specific extensions
lnvm LightNVM specific extensions
memblaze Memblaze vendor specific extensions
See 'nvme <plugin> help' for more information on a plugin
As you can see the above $ ./nvme –help command, That type of command has changed.
Especially, In NVMe-cli-ver 1.1, note that plugin extension of intel, lnvm, memlblaze.
So later On, I have to notice plugin extensions.
For example, let’s see “nvme lnvm help”
$ ./nvme lnvm help
nvme-1.1
usage: nvme lnvm <command> [<device>] [<args>]
The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).
LightNVM specific extensions
The following are all implemented sub-commands:
list List available LightNVM devices
info List general information and available target engines
id-ns List geometry for LightNVM device
init Initialize media manager on LightNVM device
create Create target on top of a LightNVM device
remove Remove target from device
factory Reset device to factory state
diag-bbtbl Diagnose bad block table
diag-set-bbtbl Update bad block table
version Shows the program version
help Display this help
See 'nvme lnvm help <command>' for more information on a specific command
As you can see the above output, note that lnvm-nvme.c and lnvm-nvme.h files.
as you note those files, lnvm-nvme.c and lnvm-nvme.h.
the total pairs for plugin exist like this :
-
intel-nvme.c and intel-nvme.h
-
lnvm-nvme.c and lnvm-nvme.h
-
memblaze-nvme.c and memblaze-nvme.h
BUT I think the pair of files that I have to research a little more exist like this :
-
nvme-lightnvm.c and nvme-lightnvm.h
-
fabrics.c and fabrics.h
-
json.c and json.h
-
nvme-models.c and nvme-models.h
-
plugin.c and plugin.h
and so on.
how to make VU command In NVMe-cli tool
First, NVMe-cli ver 0.7
you have to change files below
At first, I look at files on NVMe-cli ver 0.7
As you could see before, file of NVMe-clie ver 0.7 is the following :
$ cd nvme-cli--0.7
$ ls -v
CONTRIBUTING.md NVME-VERSION-GEN linux nvme-ioctl.c regress
Documentation README.md nvme.c nvme-ioctl.h scripts
LICENSE completions nvme.control.in nvme-print.c src
Makefile debian nvme.spec.in nvme-print.h
$ cd linux
$ ls -v
nvme.h
I changed files for VU command like this :
In linux directory.
-
nvme.h : what I change is because I have to add opcode(appointment between host and firmware),
In other words, firmware perceives what kind of command host sent to firmware through opcode.
In nvme-cli–0.7 directory
-
nvme.c : this file includes the main functions. So I have to chage this file to add VU command with
-
nvme-print.c and nvme-print.h : this file includes how to print with return value from firmware,
-
nvme-ioctl.c and nvme-ioctl.h : basic way to use IOCTL in kernel is included in this file. So You have to change this file to use the suitable IOCTL to fit firmware
let’s make a example with virtual VU command.
Each files of NVMe-cli ver 0.7 that I mentioned.
if There is a functions for VU command, that function is used to have converstion with firmware to get information to make from Nand.
And with the function, Vendors determine each commands have different opcodes for conversation with firmware
let’s suppose the function name for VU command is __descriptor__ and the name of a VU command is __get-capability__
/nvme-cli-0.7/linux/nvme.h
In my case, I decided I should add opcodes for function and command into this file.
First opcode of Admin commands
// In /nvme-cli-0.7.linux/nvme.h
367 enum nvme_admin_opcode {
.......
384 nvme_admin_security_recv = 0x82,
385 //function opcode of VU command
386 nvme_admin_descriptor = 0x00,
391 };
// AND
// The following structure I made for opcode of VU command
556 enum vu_opcode {
557 // OPCODE of VU command
564 VU_OPCODE_GET_CAPABILITY = 0x01, // get capability
.......
569 };
As you can see the above code of an example. I made opcode in /nvme-cli-0.7/linux/nvme.h to manage it easily.
/nvme-cli-0.7/nvme-print.h and /nvme-cli-0.7/nvme-print.c
I just wanted to maintain the original state of nvme-cli-0.7. In order to keep structure of opensource(nvme-cli-0.7), I analyzed the nvme-cli-0.7.
And then, I realized all print function was in this files.
So, I used this files for print functions of VU commands.
let’s see an example code
you can see how to handle the return value from Nand, after issuing VU command.
// In /nvme-cli-0.7/nvme-print.h
// this is just changed for header file of nvme.c
// So, I just declare the function.
11
12 #ifndef COMMON_H
13 #define COMMON_H
14
15 #include "linux/nvme.h"
16
17 enum {
18 TERSE = 0x1u, // only show a few useful fields
19 HUMAN = 0x2u, // interpret some values for humans
20 VS = 0x4u, // print vendor specific data area
21 RAW = 0x8u, // just dump raw bytes
22 };
23
.......
33 void show_smart_log(struct nvme_smart_log *smart, unsigned int nsid, const char *devname);
34 void show_fw_log(struct nvme_firmware_log_page *fw_log, const char *devname);
.......
41 /* print function for VU commad */
42
43 void nvme_show_capability(__u32 *vu_desc);
.......
45 #endif
// In /nvme-cli-0.7/nvme-print.c
// you can see the whole function of print.
1121 // output of get_capability opcode
1122 void nvme_show_capability(__u32 *vu_desc)
1123 {
1124
1125 printf("result : %u\n", vu_desc[0]);
1126 printf("num channels : %u\n", vu_desc[1]);
.......
1147 }
1148
/nvme-cli-0.7/nvme-ioctl.h and /nvme-cli-0.7/nvme-ioctl.c
these files include the main functions of ioctl
let’ see how to add ioctl for VU command
// In /nvme-cli-0.7/nvme-ioctl.c
// you can see what is header file of nvme-ioctl.c
12 #ifndef _NVME_LIB_H
13 #define _NVME_LIB_H
14
15 #include <stdbool.h>
16 #include "linux/nvme.h"
17
18 int nvme_get_nsid(int fd);
19
20 /* Generic passthrough */
21 int nvme_submit_passthru(int fd, int ioctl_cmd, struct nvme_passthru_cmd *cmd);
22
.......
29
30 /* NVME_SUBMIT_IO */
31 int nvme_io(int fd, __u8 opcode, __u64 slba, __u16 nblocks, __u16 control,
32 __u32 dsmgmt, __u32 reftag, __u16 apptag,
33 __u16 appmask, void *data, void *metadata);
34
.......
128 /* ioctl function for VU commad */
129
130 int descriptor(int fd, __u32 cdw10, __u32 data_len, void *data);
131 int call_descriptor(int fd, __u32 len, void *data);
132
133 #endif /* _NVME_LIB_H */
As you can see the above code, I made two functions, that is why it is easy to manange functions.
In other words, after extracting the common operation. I split multiple operation in one function into a operation in one function.
Here, I used opcode of ioctl function of VU command.
// In /nvme-cli-0.7/nvme-ioctl.c
// ioctl function of VU command
// depending on namespace, you can issue the VU command
595 int descriptor(int fd,__u32 nsid __u32 cdw10, __u32 data_len, void *data)
596 {
597 struct nvme_admin_cmd cmd = {
598 .opcode = nvme_admin_descriptor(0x00),
599 .nsid = nsid,
600 .cdw10 = cdw10,
.......
616 .addr = (__u64)(uintptr_t) data,
617 .data_len = data_len,
618 };
619
620 return nvme_submit_admin_passthru(fd, &cmd);
621 }
622
623 int call_descriptor(int fd,__u32 nsid, __u32 cdw10, __u32 len, void *data)
624 {
625 return send_descriptor(fd, nsid, cdw10, len, data);
626 }
627
As you can see the above cmd of nvme_admin_cmd. cmd is different depending on vendors.
because vendors can decide cmd property to fit vendor’s spec.
The above spec bases in SMART-LOG command.
So detail for the admin_cmd of VU command depends on Vendor.
/nvme-cli-0.7/nvme.c
this file is very important. because when typing “./nvme –help”.
the upcoming result is from this file.
In /nvme-cli-0.7
$ ./nvme --help
nvme-0.7
usage: nvme <command> [<device>] [<args>]
The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).
The following are all implemented sub-commands:
list List all NVMe devices and namespaces on machine
id-ctrl Send NVMe Identify Controller
.......
get-capability Show you capability
move-vblk Move pages from a source vblk to a destination vblk
See 'nvme help <command>' for more information on a specific command.
As you can see the above result of “./nvme –help”, you can find out get-capability sub-command.
This command, get-capability is function for VU command.
In /nvme-cli-0.7/nvme.c
78 #define COMMAND_LIST \
79 ENTRY(LIST, "list", "List all NVMe devices and namespaces on machine", list) \
80 ENTRY(ID_CTRL, "id-ctrl", "Send NVMe Identify Controller", id_ctrl) \
.......
120 ENTRY(GET_CAPABILITY, "get-capability", "Show you capability", get_capability) \
.......
127 #define ENTRY(i, n, h, f) \
In here, you can add functions and VU commands for “./nvme
In my case, I added get-capability of VU command.
120 ENTRY(GET_CAPABILITY, "get-capability", "Show you capability", get_capability) \
let’s see the actual function of get_capability in /nvme-cli-0.7/nvme.c
2793 // VU_OPCODE_GET_FLASH_CAP - Get Flash Capbility
2794 static int get_flash_capability(int argc, char **argv)
2795 {
2796 // this is for description per command.
2797 const char *desc = "Get Initial capability to manage";
2798 const char *shell_test = "this is for shell script test";
2799
2800 int err;
2801 __u32 *vu_desc = NULL; // VU command
2802 void * buf = NULL; // buffer of VU command
2803
.......
2805 struct config {
2806 __u32 vu_desc_len;
2807 __u32 shell_test;
2808 };
2809
2810 struct config cfg = {
2811 .vu_desc_len = 0,
2812 .shell_test = 0,
2813 };
2814
2815 const struct argconfig_commandline_options command_line_options[] = {
........ // for command line, "./nvme get-capability --help"
2817 {"shell_test", 'e', "NUM", CFG_POSITIVE, &cfg.shell_test, no_argument, shell_test},
2818 {0}
2819 };
2820 // this function parse arg of command, "./nvme get-capability"
2821 parse_and_open(argc, argv, desc, command_line_options, &cfg, sizeof(cfg));
2822
2823 // getpagesize() = 4096
2824 if (cfg.vu_desc_len == 0) {
2825 cfg.vu_desc_len = 512 or 4096; // depending on vendor
2826 if (posix_memalign(&buf, getpagesize(), cfg.vu_desc_len))
2827 exit(ENOMEM);
2828 memset(buf, 0, cfg.vu_desc_len);
2829 }
2830
2831 vu_desc =(__u32 *)buf;
2832
2833 vu_desc[0] = VU_OPCODE_GET_CAPABILITY(0x01);
2834
2835 // this part pointer is tranlated to
2836 err = call_descriptor(fd, cfg.vu_desc_len, buf);
2837 // if the VU command is excuted correctly.
2838 if(!err)
2839 nvme_show_capability(buf);
2840 else {
.......
2844 fprintf(stderr, "err number(hexa, deci(__u32)) : 0x%X, %u\n", err, err);
2845 fprintf(stderr, "result(hexa, deci(__u32)) : 0x%X, %u\n", vu_desc[0], vu_desc[0]);
2846 printf("get-capatilibty:Error\n");
2847 }
2848
2849 if(buf)
2850 free(buf);
2851
2852 return err;
2853 }
As you can see the above example code, the above is basic format for VU command.
let’s see VU command with –help options
$ ./nvme get-capability --help
Usage: nvme get-capability <device> [OPTIONS]
Get Initial flash capability to manage
Options:
[ --shell_test, -e ] --- this is for shell script test
So far, How to fix nvme-cli-0.7 for VU command.
Next, I will try to change nvme-cli-1.1 for VU command.
Second, NVMe-cli ver 1.1
let’s see nvme-cli-1.1 release’s file structure.
$ cd nvme-cli-1.1
$ ls -v
CONTRIBUTING.md NVME-VERSION-GEN cmd.h debian intel-nvme.c linux memblaze-nvme.h nvme.spec.in nvme-lightnvm.c nvme-print.c plugin.c suffix.c
Documentation README.md cmd_handler.h define_cmd.h intel-nvme.h lnvm-nvme.c nvme.c nvme-builtin.h nvme-lightnvm.h nvme-print.h plugin.h suffix.h
LICENSE argconfig.c common.h fabrics.c json.c lnvm-nvme.h nvme.control.in nvme-ioctl.c nvme-models.c parser.c regress tests
Makefile argconfig.h completions fabrics.h json.h memblaze-nvme.c nvme.h nvme-ioctl.h nvme-models.h parser.h scripts
$ cd linux
$ ls -v
lightnvm.h nvme.h nvme_ioctl.h
As you can see the above result, many things is different from nvme-cli-0.7 release.
BUT, If you want to make VU command in nvme-cli-1.1, it is easy under a similar way.
In other words, Basically, when you change the nvme-cli-1.1 with a similar way to nvme-cli-0.7 without plugin mode, files that you have to change is similar to nvme-cli-0.7
Without plugin mode, that is temporary way.
first, I will explain to you how to change nvme-cli-1.1 for VU command without plugin mode.
Without writting opcode in /nvme-cli-1.1/linux/nvme.h, everything is almost the same.
The files that I have to change :
-
nvme.c : this is the same from nvme-cli-0.7, this file includes function for the VU command
-
nvme-print.c and nvme-print.h : this includes function of print for VU command as well.
-
nvme-ioctl.c and nvme-ioctl.h : this includes funtion of ioctl for VU command as well.
-
nvme-builtin.h : this file is new, just on ./nvme
, this is file for
Each files of NVMe-cli ver 1.1
/nvme-cli-1.1/nvme-builtin.h
In my case, so as to register VU command, I had to change this file first.
let’s see the code.
$ vim nvme-builtin.h
// In /nvme-cli-1.1/nvme-builtin.h
// in order to register the name of command and function.
12 #undef CMD_INC_FILE
13 #define CMD_INC_FILE nvme-builtin
14
15 #if !defined(NVME_BUILTIN) || defined(CMD_HEADER_MULTI_READ)
16 #define NVME_BUILTIN
17
18 #include "cmd.h"
19
20 COMMAND_LIST(
21 ENTRY("list", "List all NVMe devices and namespaces on machine", list)
22 ENTRY("id-ctrl", "Send NVMe Identify Controller", id_ctrl)
23 ENTRY("id-ns", "Send NVMe Identify Namespace, display structure", id_ns)
24 ENTRY("list-ns", "Send NVMe Identify List, display structure", list_ns)
25 ENTRY("create-ns", "Creates a namespace with the provided parameters", create_ns)
26 ENTRY("delete-ns", "Deletes a namespace from the controller", delete_ns)
27 ENTRY("attach-ns", "Attaches a namespace to requested controller(s)", attach_ns)
28 ENTRY("detach-ns", "Detaches a namespace from requested controller(s)", detach_ns)
29 ENTRY("list-ctrl", "Send NVMe Identify Controller List, display structure", list_ctrl)
30 ENTRY("get-ns-id", "Retrieve the namespace ID of opened block device", get_ns_id)
31 ENTRY("get-log", "Generic NVMe get log, returns log in raw format", get_log)
32 ENTRY("fw-log", "Retrieve FW Log, show it", get_fw_log)
33 ENTRY("smart-log", "Retrieve SMART Log, show it", get_smart_log)
34 ENTRY("error-log", "Retrieve Error Log, show it", get_error_log)
35 ENTRY("get-feature", "Get feature and show the resulting value", get_feature)
36 ENTRY("set-feature", "Set a feature and show the resulting value", set_feature)
37 ENTRY("format", "Format namespace with new block format", format)
38 ENTRY("fw-activate", "Activate new firmware slot", fw_activate)
39 ENTRY("fw-download", "Download new firmware", fw_download)
40 ENTRY("admin-passthru", "Submit arbitrary admin command, return results", admin_passthru)
41 ENTRY("io-passthru", "Submit an arbitrary IO command, return results", io_passthru)
42 ENTRY("security-send", "Submit a Security Send command, return results", sec_send)
43 ENTRY("security-recv", "Submit a Security Receive command, return results", sec_recv)
44 ENTRY("resv-acquire", "Submit a Reservation Acquire, return results", resv_acquire)
45 ENTRY("resv-register", "Submit a Reservation Register, return results", resv_register)
46 ENTRY("resv-release", "Submit a Reservation Release, return results", resv_release)
47 ENTRY("resv-report", "Submit a Reservation Report, return results", resv_report)
48 ENTRY("dsm", "Submit a Data Set Management command, return results", dsm)
49 ENTRY("flush", "Submit a Flush command, return results", flush)
50 ENTRY("compare", "Submit a Compare command, return results", compare)
51 ENTRY("read", "Submit a read command, return results", read_cmd)
52 ENTRY("write", "Submit a write command, return results", write_cmd)
53 ENTRY("write-zeroes", "Submit a write zeroes command, return results", write_zeroes)
54 ENTRY("write-uncor", "Submit a write uncorrectable command, return results", write_uncor)
55 ENTRY("reset", "Resets the controller", reset)
56 ENTRY("subsystem-reset", "Resets the controller", subsystem_reset)
57 ENTRY("show-regs", "Shows the controller registers. Requires admin character device", show_registers)
58 ENTRY("discover", "Discover NVMeoF subsystems", discover_cmd)
59 ENTRY("connect-all", "Discover and Connect to NVMeoF subsystems", connect_all_cmd)
60 ENTRY("connect", "Connect to NVMeoF subsystem", connect_cmd)
61 ENTRY("disconnect", "Disconnect from NVMeoF subsystem", disconnect_cmd)
62 // For VU command
63 ENTRY("get-information", "Get information from NVMe subsystem",get_information)
64 );
65
66 #endif
67
68 #include "define_cmd.h"
As you can see the above code, You can check the above contents with “./nvme –help”
$ ./nvme --help
nvme-1.1
usage: nvme <command> [<device>] [<args>]
The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).
The following are all implemented sub-commands:
list List all NVMe devices and namespaces on machine
id-ctrl Send NVMe Identify Controller
id-ns Send NVMe Identify Namespace, display structure
list-ns Send NVMe Identify List, display structure
create-ns Creates a namespace with the provided parameters
delete-ns Deletes a namespace from the controller
attach-ns Attaches a namespace to requested controller(s)
detach-ns Detaches a namespace from requested controller(s)
list-ctrl Send NVMe Identify Controller List, display structure
get-ns-id Retrieve the namespace ID of opened block device
get-log Generic NVMe get log, returns log in raw format
fw-log Retrieve FW Log, show it
smart-log Retrieve SMART Log, show it
error-log Retrieve Error Log, show it
get-feature Get feature and show the resulting value
set-feature Set a feature and show the resulting value
format Format namespace with new block format
fw-activate Activate new firmware slot
fw-download Download new firmware
admin-passthru Submit arbitrary admin command, return results
io-passthru Submit an arbitrary IO command, return results
security-send Submit a Security Send command, return results
security-recv Submit a Security Receive command, return results
resv-acquire Submit a Reservation Acquire, return results
resv-register Submit a Reservation Register, return results
resv-release Submit a Reservation Release, return results
resv-report Submit a Reservation Report, return results
dsm Submit a Data Set Management command, return results
flush Submit a Flush command, return results
compare Submit a Compare command, return results
read Submit a read command, return results
write Submit a write command, return results
write-zeroes Submit a write zeroes command, return results
write-uncor Submit a write uncorrectable command, return results
reset Resets the controller
subsystem-reset Resets the controller
show-regs Shows the controller registers. Requires admin character device
discover Discover NVMeoF subsystems
connect-all Discover and Connect to NVMeoF subsystems
connect Connect to NVMeoF subsystem
disconnect Disconnect from NVMeoF subsystem
get-information Get information from NVMe subsystem
version Shows the program version
help Display this help
.......
you could see new command, get-information, additionally.
/nvme-cli-1.1/nvme-print.c and /nvme-cli-1.1/nvme-print.h
In my case, I added function of print for get-information of VU command.
First of all, let’s see the /nvme-cli-1.1/nvme-print.h
// In /nvme-cli-1.1/nvme-print.h
13 #ifndef NVME_PRINT_H
14 #define NVME_PRINT_H
15
16 #include "nvme.h"
17 #include <inttypes.h>
18
19 enum {
20 TERSE = 0x1u, // only show a few useful fields
21 HUMAN = 0x2u, // interpret some values for humans
22 VS = 0x4u, // print vendor specific data area
23 RAW = 0x8u, // just dump raw bytes
24 };
25
26 void d(unsigned char *buf, int len, int width, int group);
27 void d_raw(unsigned char *buf, unsigned len);
28
29 uint64_t int48_to_long(__u8 *data);
30
31 void __show_nvme_id_ctrl(struct nvme_id_ctrl *ctrl, unsigned int mode, void (*vendor_show)(__u8 *vs));
32 void show_nvme_id_ctrl(struct nvme_id_ctrl *ctrl, unsigned int mode);
.......
57 //For VU command
58 void show_information(__u32 *result, unsigned int nsid, const char *devname);
59
60
61 #endif
this file has role as header file.
In /nvme-cli-1.1/nvme-print.c
1791 // For VU command of SK hynix SSD
1792 void show_information(__u32 *result, unsigned int nsid, const char *devname)
1793 {
1794 printf("information NVME device:%s namespace-id:%x\n", devname, nsid);
1795 printf("dw[0] : %u, 0x%x\n", result[0], result[0]);
.........
1802 printf("dw[3] : %u, 0x%x\n", result[7], result[7]);
1803 }
1804
on the above function, with buffer between host and firmware.
you can print the return value.
/nvme-cli-1.1/nvme-ioctl.c and /nvme-cli-1.1/nvme-ioctl.h
this is also the same from nvme-cli-0.7’s nvme-ioctol file.
In /nvme-cli-1.1/nvme-ioctl.h
11 #ifndef _NVME_LIB_H
12 #define _NVME_LIB_H
13
14 #include <linux/types.h>
15 #include <stdbool.h>
16 #include "linux/nvme_ioctl.h"
17 #include "nvme.h"
18
19 int nvme_get_nsid(int fd);
20
21 /* Generic passthrough */
22 int nvme_submit_passthru(int fd, int ioctl_cmd, struct nvme_passthru_cmd *cmd);
23
24 int nvme_passthru(int fd, int ioctl_cmd, __u8 opcode, __u8 flags,
25 __u16 rsvd, __u32 nsid, __u32 cdw2, __u32 cdw3,
26 __u32 cdw10, __u32 cdw11, __u32 cdw12,
27 __u32 cdw13, __u32 cdw14, __u32 cdw15,
28 __u32 data_len, void *data, __u32 metadata_len,
29 void *metadata, __u32 timeout_ms, __u32 *result);
30
31 /* NVME_SUBMIT_IO */
32 int nvme_io(int fd, __u8 opcode, __u64 slba, __u16 nblocks, __u16 control,
33 __u32 dsmgmt, __u32 reftag, __u16 apptag,
34 __u16 appmask, void *data, void *metadata);
.........
127 int nvme_subsystem_reset(int fd);
128 int nvme_reset_controller(int fd);
129 /*For VU command*/
130
131 int descriptor(int fd, __u32 nsid, __u32 cdw10, __u32 data_len, void *data);
132 int nvme_information(int fd, __u32 nsid, __u32 len, void *data);
133
134
135 #endif /* _NVME_LIB_H */
This file is also similar to /nvme-cli-0.7/nvme-ioctl.h
it’s just different from function name.
In /nvme-cli-1.1/nvme-ioctl.c
605 // For VU command
606
607 int descriptor(int fd, __u32 nsid, __u32 cdw10, __u32 data_len, void *data)
608 {
609 struct nvme_admin_cmd cmd = {
610 .opcode = 0x00,//nvme_admin_descriptor;
611 .nsid = nsid,
612 .cdw10 = cdw10,
613 .addr = (__u64)(uintptr_t) data,
614 .data_len = data_len,
615 };
616
617 return nvme_submit_admin_passthru(fd, &cmd);
618 }
619
620
621 int nvme_information(int fd, __u32 nsid, __u32 len, void *data)
622 {
623 return descriptor(fd, nsid, len, len, data);
624 }
The above example code is also based on smart-log of nvme-cli-1.1.
let’s last see VU command function.
/nvme-cli-1.1/nvme.c
In this file, I used get_smart_log function of get_inforamtion that is virtual VU command.
184 // FOR VU command
185 // get_information
186 static int get_information(int argc, char **argv, struct command *cmd, struct plugin *plugin)
187 {
188 const char *desc = "Retrieve information for the given device "\
189 "(or optionally a namespace) in either decoded format "\
190 "(default) or binary.";
191 const char *namespace = "(optional) desired namespace";
192 const char *raw = "output in binary format";
193 int err, fmt, fd;
194 __u32 *vu_desc = NULL; // VU command
195 void *buf = NULL; // Buffer for VU command
196
197 struct config {
198 __u32 vu_desc_len;
199 __u32 namespace_id;
200 int raw_binary;
201 char *output_format;
202 };
203
204 struct config cfg = {
205 .vu_desc_len = 0,
206 .namespace_id = 0xffffffff,
207 .output_format = "normal",
208 };
209
210 const struct argconfig_commandline_options command_line_options[] = {
211 {"namespace-id", 'n', "NUM", CFG_POSITIVE, &cfg.namespace_id, required_argument, namespace},
212 {"output-format", 'o', "FMT", CFG_STRING, &cfg.output_format, required_argument, output_format },
213 {"raw-binary", 'b', "", CFG_NONE, &cfg.raw_binary, no_argument, raw},
214 {NULL}
215 };
216
217 fd = parse_and_open(argc, argv, desc, command_line_options, &cfg, sizeof(cfg));
218 if (fd < 0)
219 return fd;
220
221 fmt = validate_output_format(cfg.output_format);
222 if (fmt < 0)
223 return fmt;
224 if (cfg.raw_binary)
225 fmt = BINARY;
226
227 // getpagesize() = 4096
228 if (cfg.vu_desc_len == 0){
229 cfg.vu_desc_len = 4096 or 512; depending on vendors
230 if (posix_memalign(&buf, getpagesize(), cfg.vu_desc_len))
231 exit(ENOMEM);
232 memset(buf, 0, cfg.vu_desc_len);
233 }
234
235 vu_desc = (__u32 *)buf;
236 vu_desc[0] = 0x01; //VU_OPCODE_GET_INFORMATION;
237
238
239 err = nvme_information(fd, cfg.namespace_id, cfg.vu_desc_len, buf);
240 if (!err) {
241 if (fmt == BINARY)
242 fprintf(stderr, "this does not support Binary mode\n");
243 else if (fmt == JSON)
244 fprintf(stderr, "this does not support json mode\n");
245 else
246 show_information(buf, cfg.namespace_id, devicename);
247 }
248 else if (err > 0)
249 fprintf(stderr, "NVMe Status:%s(%x)\n",
250 nvme_status_to_string(err), err);
251 else
252 perror("get information error");
253
254 if (buf != 0)
255 free(buf);
256
257 return err;
258 }
let’s see output of help option of get-information command
In /nvme-cli-1.1
$ ./nvme get-information --help
Usage: nvme get-information <device> [OPTIONS]
Retrieve information for the given device (or optionally a namespace) in
either decoded format (default) or binary.
Options:
[ --namespace-id=<NUM>, -n <NUM> ] --- (optional) desired namespace
[ --output-format=<FMT>, -o <FMT> ] --- Output format: normal|json|binary
[ --raw-binary, -b ] --- output in binary format
Appendix A
At first, nvme-builtin.h
#undef CMD_INC_FILE
#define CMD_INC_FILE nvme-builtin
#if !defined(NVME_BUILTIN) || defined(CMD_HEADER_MULTI_READ)
#define NVME_BUILTIN
#include "cmd.h"
COMMAND_LIST(
ENTRY("list", "List all NVMe devices and namespaces on machine", list)
ENTRY("id-ctrl", "Send NVMe Identify Controller", id_ctrl)
ENTRY("id-ns", "Send NVMe Identify Namespace, display structure", id_ns)
ENTRY("list-ns", "Send NVMe Identify List, display structure", list_ns)
ENTRY("create-ns", "Creates a namespace with the provided parameters", create_ns)
ENTRY("delete-ns", "Deletes a namespace from the controller", delete_ns)
ENTRY("attach-ns", "Attaches a namespace to requested controller(s)", attach_ns)
ENTRY("detach-ns", "Detaches a namespace from requested controller(s)", detach_ns)
ENTRY("list-ctrl", "Send NVMe Identify Controller List, display structure", list_ctrl)
ENTRY("get-ns-id", "Retrieve the namespace ID of opened block device", get_ns_id)
ENTRY("get-log", "Generic NVMe get log, returns log in raw format", get_log)
ENTRY("fw-log", "Retrieve FW Log, show it", get_fw_log)
ENTRY("smart-log", "Retrieve SMART Log, show it", get_smart_log)
ENTRY("error-log", "Retrieve Error Log, show it", get_error_log)
ENTRY("get-feature", "Get feature and show the resulting value", get_feature)
ENTRY("set-feature", "Set a feature and show the resulting value", set_feature)
ENTRY("format", "Format namespace with new block format", format)
ENTRY("fw-activate", "Activate new firmware slot", fw_activate)
ENTRY("fw-download", "Download new firmware", fw_download)
ENTRY("admin-passthru", "Submit arbitrary admin command, return results", admin_passthru)
ENTRY("io-passthru", "Submit an arbitrary IO command, return results", io_passthru)
ENTRY("security-send", "Submit a Security Send command, return results", sec_send)
ENTRY("security-recv", "Submit a Security Receive command, return results", sec_recv)
ENTRY("resv-acquire", "Submit a Reservation Acquire, return results", resv_acquire)
ENTRY("resv-register", "Submit a Reservation Register, return results", resv_register)
ENTRY("resv-release", "Submit a Reservation Release, return results", resv_release)
ENTRY("resv-report", "Submit a Reservation Report, return results", resv_report)
ENTRY("dsm", "Submit a Data Set Management command, return results", dsm)
ENTRY("flush", "Submit a Flush command, return results", flush)
ENTRY("compare", "Submit a Compare command, return results", compare)
ENTRY("read", "Submit a read command, return results", read_cmd)
ENTRY("write", "Submit a write command, return results", write_cmd)
ENTRY("write-zeroes", "Submit a write zeroes command, return results", write_zeroes)
ENTRY("write-uncor", "Submit a write uncorrectable command, return results", write_uncor)
ENTRY("reset", "Resets the controller", reset)
ENTRY("subsystem-reset", "Resets the controller", subsystem_reset)
ENTRY("show-regs", "Shows the controller registers. Requires admin character device", show_registers)
ENTRY("discover", "Discover NVMeoF subsystems", discover_cmd)
ENTRY("connect-all", "Discover and Connect to NVMeoF subsystems", connect_all_cmd)
ENTRY("connect", "Connect to NVMeoF subsystem", connect_cmd)
ENTRY("disconnect", "Disconnect from NVMeoF subsystem", disconnect_cmd)
);
#endif
#include "define_cmd.h"
Second, nvme.c
static int get_smart_log(int argc, char **argv, struct command *cmd, struct plugin *plugin)
{
struct nvme_smart_log smart_log;
const char *desc = "Retrieve SMART log for the given device "\
"(or optionally a namespace) in either decoded format "\
"(default) or binary.";
const char *namespace = "(optional) desired namespace";
const char *raw = "output in binary format";
int err, fmt, fd;
struct config {
__u32 namespace_id;
int raw_binary;
char *output_format;
};
struct config cfg = {
.namespace_id = 0xffffffff,
.output_format = "normal",
};
const struct argconfig_commandline_options command_line_options[] = {
{"namespace-id", 'n', "NUM", CFG_POSITIVE, &cfg.namespace_id, required_argument, namespace},
{"output-format", 'o', "FMT", CFG_STRING, &cfg.output_format, required_argument, output_format },
{"raw-binary", 'b', "", CFG_NONE, &cfg.raw_binary, no_argument, raw},
{NULL}
};
fd = parse_and_open(argc, argv, desc, command_line_options, &cfg, sizeof(cfg));
if (fd < 0)
return fd;
fmt = validate_output_format(cfg.output_format);
if (fmt < 0)
return fmt;
if (cfg.raw_binary)
fmt = BINARY;
err = nvme_smart_log(fd, cfg.namespace_id, &smart_log);
if (!err) {
if (fmt == BINARY)
d_raw((unsigned char *)&smart_log, sizeof(smart_log));
else if (fmt == JSON)
json_smart_log(&smart_log, cfg.namespace_id, devicename);
else
show_smart_log(&smart_log, cfg.namespace_id, devicename);
}
else if (err > 0)
fprintf(stderr, "NVMe Status:%s(%x)\n",
nvme_status_to_string(err), err);
else
perror("smart log");
return err;
}
From now on, just function to print
Third, nvme-print.h
#ifndef NVME_PRINT_H
#define NVME_PRINT_H
#include "nvme.h"
#include <inttypes.h>
enum {
TERSE = 0x1u, // only show a few useful fields
HUMAN = 0x2u, // interpret some values for humans
VS = 0x4u, // print vendor specific data area
RAW = 0x8u, // just dump raw bytes
};
void d(unsigned char *buf, int len, int width, int group);
void d_raw(unsigned char *buf, unsigned len);
uint64_t int48_to_long(__u8 *data);
void __show_nvme_id_ctrl(struct nvme_id_ctrl *ctrl, unsigned int mode, void (*vendor_show)(__u8 *vs));
void show_nvme_id_ctrl(struct nvme_id_ctrl *ctrl, unsigned int mode);
void show_nvme_id_ns(struct nvme_id_ns *ns, unsigned int flags);
void show_nvme_resv_report(struct nvme_reservation_status *status);
void show_lba_range(struct nvme_lba_range_type *lbrt, int nr_ranges);
void show_error_log(struct nvme_error_log_page *err_log, int entries, const char *devname);
void show_intel_smart_log(struct nvme_additional_smart_log *smart, unsigned int nsid, const char *devname);
void show_smart_log(struct nvme_smart_log *smart, unsigned int nsid, const char *devname);
void show_fw_log(struct nvme_firmware_log_page *fw_log, const char *devname);
void show_ctrl_registers(void *bar, unsigned int mode);
void nvme_feature_show_fields(__u32 fid, unsigned int result, unsigned char *buf);
char *nvme_status_to_string(__u32 status);
char *nvme_select_to_string(int sel);
char *nvme_feature_to_string(int feature);
void json_nvme_id_ctrl(struct nvme_id_ctrl *ctrl, unsigned int mode);
void json_nvme_id_ns(struct nvme_id_ns *ns, unsigned int flags);
void json_nvme_resv_report(struct nvme_reservation_status *status);
void json_error_log(struct nvme_error_log_page *err_log, int entries, const char *devname);
void json_smart_log(struct nvme_smart_log *smart, unsigned int nsid, const char *devname);
void json_add_smart_log(struct nvme_additional_smart_log *smart,
unsigned int nsid, const char *devname);
void json_fw_log(struct nvme_firmware_log_page *fw_log, const char *devname);
void json_print_list_items(struct list_item *items, unsigned amnt);
#endif
Addtionally, nvme-print.c file
finally, IF you change this file. you can make nvme-cli tool to support VU command.
be careful, what I am saying is what version of NVMe-cli, because later on, version will be changed.
BUT, If you want to support man operation of linux. you need a little more
let’s make a code for VU command
Ahead of making VU command, let’s check what version of nvme-cli is right after “git clone https://github.com/linux-nvme/nvme-cli.git”
in nvme directory, type ./nvme after that, you would get the following result.
$ ./nvme
nvme-1.1.2.g0142
usage: nvme <command> [<device>] [<args>]
The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).
The following are all implemented sub-commands:
list List all NVMe devices and namespaces on machine
id-ctrl Send NVMe Identify Controller
id-ns Send NVMe Identify Namespace, display structure
list-ns Send NVMe Identify List, display structure
create-ns Creates a namespace with the provided parameters
delete-ns Deletes a namespace from the controller
attach-ns Attaches a namespace to requested controller(s)
detach-ns Detaches a namespace from requested controller(s)
list-ctrl Send NVMe Identify Controller List, display structure
get-ns-id Retrieve the namespace ID of opened block device
get-log Generic NVMe get log, returns log in raw format
fw-log Retrieve FW Log, show it
smart-log Retrieve SMART Log, show it
error-log Retrieve Error Log, show it
get-feature Get feature and show the resulting value
set-feature Set a feature and show the resulting value
format Format namespace with new block format
fw-activate Activate new firmware slot
fw-download Download new firmware
admin-passthru Submit arbitrary admin command, return results
io-passthru Submit an arbitrary IO command, return results
security-send Submit a Security Send command, return results
security-recv Submit a Security Receive command, return results
resv-acquire Submit a Reservation Acquire, return results
resv-register Submit a Reservation Register, return results
resv-release Submit a Reservation Release, return results
resv-report Submit a Reservation Report, return results
dsm Submit a Data Set Management command, return results
flush Submit a Flush command, return results
compare Submit a Compare command, return results
read Submit a read command, return results
write Submit a write command, return results
write-zeroes Submit a write zeroes command, return results
write-uncor Submit a write uncorrectable command, return results
reset Resets the controller
subsystem-reset Resets the controller
show-regs Shows the controller registers. Requires admin character device
discover Discover NVMeoF subsystems
connect-all Discover and Connect to NVMeoF subsystems
connect Connect to NVMeoF subsystem
disconnect Disconnect from NVMeoF subsystem
version Shows the program version
help Display this help
See 'nvme help <command>' for more information on a specific command
The following are all installed plugin extensions:
intel Intel vendor specific extensions
lnvm LightNVM specific extensions
memblaze Memblaze vendor specific extensions
See 'nvme <plugin> help' for more information on a plugin
As you can see below, the version of NVMe-cli is nvme-1.1.2.g0142.
So I could determine What version I would make VU command on.
I decided That version is 1.1, you can get download in release of the officail site of nvme-cli.
here is v1.1 of nvme-cli release.
let’s start making VU command on NVMe-cli 1.1 release
First, You have to download nvme-cli release version 1.1.
In my case, I downloaded nvme-cli-1.1.zip
let’s uncompress that file.
$ unzip nvme-cli-1.1.zip
After that, as you can see the list with ls command, you would find out nvme-cli-1.1
$ ls
nvme-cli-0.7 nvme-cli-0.7.tar nvme-cli-1.1
And then you need to chang some files to insert VU command into opensource(nvme-cli)
as you read the above files,first, I will change the file, nvme-builtin.h.
nvme-builtin.h file
above all, I would exlain to you about code of nvme-builtin.h
#undef CMD_INC_FILE
#define CMD_INC_FILE nvme-builtin
#if !defined(NVME_BUILTIN) || defined(CMD_HEADER_MULTI_READ)
#define NVME_BUILTIN
#include "cmd.h"
COMMAND_LIST(
ENTRY("list", "List all NVMe devices and namespaces on machine", list)
......
);
#endif
#include "define_cmd.h"
As you see the above code. ENTRY indicates the list of ./nvme command.
“list” - command name
“List all NVMe device and namespaces on machine” - summary about what role of this command is.
list - function name in nvme.c file
you can check up what I am saying to you indirectly. type ./nvme in nvme-cli directory.
$ ./nvme
nvme-1.1.2.g0142
usage: nvme <command> [<device>] [<args>]
The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).
The following are all implemented sub-commands:
list List all NVMe devices and namespaces on machine
....
See 'nvme help <command>' for more information on a specific command
The following are all installed plugin extensions:
intel Intel vendor specific extensions
lnvm LightNVM specific extensions
memblaze Memblaze vendor specific extensions
So you change this file like this
```c
#undef CMD_INC_FILE
#define CMD_INC_FILE nvme-builtin
#if !defined(NVME_BUILTIN) || defined(CMD_HEADER_MULTI_READ)
#define NVME_BUILTIN
#include "cmd.h"
COMMAND_LIST(
ENTRY("list", "List all NVMe devices and namespaces on machine", list)
......
// add ENTRY of VU command In here
ENTRY("command of nvme-clie for VU command", "Description of NVMe-cli command", "Function name of nvme.c file for this Command")
// For example
ENTRY("nvme-waf", "get WAF from NVMe subsystem", get_waf)
);
#endif
#include "define_cmd.h"
.......
nvme.c and nvme.h
–>