NVMe-Cli

Now, I’m looking at commit version,“0142aebd1125ae33cdf60fa957a8bfef5851a76e”, on master brach,

This tool made of OpenSource is for NVMe device.

You can do anything of command of NVMexpression specification 1.2 and so on.

it’s like this

$ sudo ./nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     SN IS ~~~~~~~~       Model's name of SSD                      1         1000.00  GB / 1000.00  GB   512   B +  0 B  Ver of FW

OR

$ sudo ./nvme smart-log /dev/nvme0 
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning                    : 0
temperature                         : 00 C
available_spare                     : 000%
available_spare_threshold           : 00%
percentage_used                     : 0%
data_units_read                     : 00
data_units_written                  : 00,000,000
host_read_commands                  : 000
host_write_commands                 : 0,000,000,000
controller_busy_time                : 000
power_cycles                        : 0
power_on_hours                      : 000
unsafe_shutdowns                    : 0
media_errors                        : 0
num_err_log_entries                 : 0
Warning Temperature Time            : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1                : 0 C
Temperature Sensor 2                : 0 C
Temperature Sensor 3                : 0 C
Temperature Sensor 4                : 0 C
Temperature Sensor 5                : 0 C
Temperature Sensor 6                : 0 C
Temperature Sensor 7                : 0 C
Temperature Sensor 8                : 0 C

As you can see the above command of nvme-cli, you can get information of nvme device with NVMe-cli tool.

So far, The above nvme-cli from the commit version is similar to NVMe-cli ver1.1 release.

Later on, I would make VU command with NVMe-cli ver 1.1 opensource based on experience with NVMe-cli ver 0.7

In other words, according to NVMe specification, The common command of all NVMe device is included in NVMe-cil tool.

Also, You can tune the NVMe-cli of opensource to make the fit VU command in addition.

let’s see NVME-Cli ver 1.1 release

NVMe-Cli ver 1.1 release

NVMe-Cli-ver 0.7

$ cd nvme-cli--0.7
$ ls -v
CONTRIBUTING.md  NVME-VERSION-GEN  linux            nvme-ioctl.c  regress
Documentation    README.md         nvme.c           nvme-ioctl.h  scripts
LICENSE          completions       nvme.control.in  nvme-print.c  src
Makefile         debian            nvme.spec.in     nvme-print.h
$ cd linux 
$ ls -v
nvme.h

and

NVMe-Cli-ver 1.1

$ cd nvme-cli-1.1
$ ls -v
CONTRIBUTING.md  NVME-VERSION-GEN  cmd.h          debian        intel-nvme.c  linux            memblaze-nvme.h  nvme.spec.in    nvme-lightnvm.c  nvme-print.c  plugin.c  suffix.c
Documentation    README.md         cmd_handler.h  define_cmd.h  intel-nvme.h  lnvm-nvme.c      nvme.c           nvme-builtin.h  nvme-lightnvm.h  nvme-print.h  plugin.h  suffix.h
LICENSE          argconfig.c       common.h       fabrics.c     json.c        lnvm-nvme.h      nvme.control.in  nvme-ioctl.c    nvme-models.c    parser.c      regress   tests
Makefile         argconfig.h       completions    fabrics.h     json.h        memblaze-nvme.c  nvme.h           nvme-ioctl.h    nvme-models.h    parser.h      scripts
$ cd linux
$ ls -v
lightnvm.h  nvme.h  nvme_ioctl.h

As you can compare NVMe-cli ver 1.1 with ver 0.7, many things is changed in NVMe-cli tool.

I think because of change of mainline of kernel.

Now, The mainline kernel supports NVMe fabric and LightNVM(Open-ChannelSSD).

So owing to NVMe Fabric, NVMe-cli tool supports json.c and fabrics.c and so on.

And you can see lnvm-nvme.c and lightnvm.h of linux directory as well.

You can notice the trend of NVMe device changes to lightNVM and Fabrics.

Also, as you can see lnvm-nvme.c file. I think the big trend of NVMe is Open-ChannelSSD and lightNVM.

./nvme between ver 1.1 and ver 0.7 of NVMe-cli

NVMe-Cli-ver 0.7

$ ./nvme --help
nvme-0.7
usage: nvme <command> [<device>] [<args>]

The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).

The following are all implemented sub-commands:
  list            List all NVMe devices and namespaces on machine
  id-ctrl         Send NVMe Identify Controller
  id-ns           Send NVMe Identify Namespace, display structure
  list-ns         Send NVMe Identify List, display structure
  create-ns       Creates a namespace with the provided parameters
  delete-ns       Deletes a namespace from the controller
  attach-ns       Attaches a namespace to requested controller(s)
  detach-ns       Detaches a namespace from requested controller(s)
  list-ctrl       Send NVMe Identify Controller List, display structure
  get-ns-id       Retrieve the namespace ID of opened block device
  get-log         Generic NVMe get log, returns log in raw format
  fw-log          Retrieve FW Log, show it
  smart-log       Retrieve SMART Log, show it
  smart-log-add   Retrieve additional SMART Log, show it
  error-log       Retrieve Error Log, show it
  get-feature     Get feature and show the resulting value
  set-feature     Set a feature and show the resulting value
  format          Format namespace with new block format
  fw-activate     Activate new firmware slot
  fw-download     Download new firmware
  admin-passthru  Submit arbitrary admin command, return results
  io-passthru     Submit an arbitrary IO command, return results
  security-send   Submit a Security Send command, return results
  security-recv   Submit a Security Receive command, return results
  resv-acquire    Submit a Reservation Acquire, return results
  resv-register   Submit a Reservation Register, return results
  resv-release    Submit a Reservation Release, return results
  resv-report     Submit a Reservation Report, return results
  dsm             Submit a Data Set Management command, return results
  flush           Submit a Flush command, return results
  compare         Submit a Compare command, return results
  read            Submit a read command, return results
  write           Submit a write command, return results
  write-zeroes    Submit a write zeroes command, return results
  write-uncor     Submit a write uncorrectable command, return results
  reset           Resets the controller
  subsystem-reset Resets the controller
  show-regs       Shows the controller registers. Requires admin character device
  version         Shows the program version
  help            Display this help

See 'nvme help <command>' for more information on a specific command.

NVMe-Cli-ver 1.1

$ ./nvme --help
nvme-1.1
usage: nvme <command> [<device>] [<args>]

The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).

The following are all implemented sub-commands:
  list            List all NVMe devices and namespaces on machine
  id-ctrl         Send NVMe Identify Controller
  id-ns           Send NVMe Identify Namespace, display structure
  list-ns         Send NVMe Identify List, display structure
  create-ns       Creates a namespace with the provided parameters
  delete-ns       Deletes a namespace from the controller
  attach-ns       Attaches a namespace to requested controller(s)
  detach-ns       Detaches a namespace from requested controller(s)
  list-ctrl       Send NVMe Identify Controller List, display structure
  get-ns-id       Retrieve the namespace ID of opened block device
  get-log         Generic NVMe get log, returns log in raw format
  fw-log          Retrieve FW Log, show it
  smart-log       Retrieve SMART Log, show it
  error-log       Retrieve Error Log, show it
  get-feature     Get feature and show the resulting value
  set-feature     Set a feature and show the resulting value
  format          Format namespace with new block format
  fw-activate     Activate new firmware slot
  fw-download     Download new firmware
  admin-passthru  Submit arbitrary admin command, return results
  io-passthru     Submit an arbitrary IO command, return results
  security-send   Submit a Security Send command, return results
  security-recv   Submit a Security Receive command, return results
  resv-acquire    Submit a Reservation Acquire, return results
  resv-register   Submit a Reservation Register, return results
  resv-release    Submit a Reservation Release, return results
  resv-report     Submit a Reservation Report, return results
  dsm             Submit a Data Set Management command, return results
  flush           Submit a Flush command, return results
  compare         Submit a Compare command, return results
  read            Submit a read command, return results
  write           Submit a write command, return results
  write-zeroes    Submit a write zeroes command, return results
  write-uncor     Submit a write uncorrectable command, return results
  reset           Resets the controller
  subsystem-reset Resets the controller
  show-regs       Shows the controller registers. Requires admin character device
  discover        Discover NVMeoF subsystems
  connect-all     Discover and Connect to NVMeoF subsystems
  connect         Connect to NVMeoF subsystem
  disconnect      Disconnect from NVMeoF subsystem
  version         Shows the program version
  help            Display this help

See 'nvme help <command>' for more information on a specific command

The following are all installed plugin extensions:
  intel           Intel vendor specific extensions
  lnvm            LightNVM specific extensions
  memblaze        Memblaze vendor specific extensions

See 'nvme <plugin> help' for more information on a plugin

As you can see the above $ ./nvme –help command, That type of command has changed.

Especially, In NVMe-cli-ver 1.1, note that plugin extension of intel, lnvm, memlblaze.

So later On, I have to notice plugin extensions.

For example, let’s see “nvme lnvm help”

$ ./nvme lnvm help    
nvme-1.1
usage: nvme lnvm <command> [<device>] [<args>]

The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).

LightNVM specific extensions

The following are all implemented sub-commands:
  list            List available LightNVM devices
  info            List general information and available target engines
  id-ns           List geometry for LightNVM device
  init            Initialize media manager on LightNVM device
  create          Create target on top of a LightNVM device
  remove          Remove target from device
  factory         Reset device to factory state
  diag-bbtbl      Diagnose bad block table
  diag-set-bbtbl  Update bad block table
  version         Shows the program version
  help            Display this help

See 'nvme lnvm help <command>' for more information on a specific command

As you can see the above output, note that lnvm-nvme.c and lnvm-nvme.h files.

as you note those files, lnvm-nvme.c and lnvm-nvme.h.

the total pairs for plugin exist like this :

  • intel-nvme.c and intel-nvme.h

  • lnvm-nvme.c and lnvm-nvme.h

  • memblaze-nvme.c and memblaze-nvme.h

BUT I think the pair of files that I have to research a little more exist like this :

  • nvme-lightnvm.c and nvme-lightnvm.h

  • fabrics.c and fabrics.h

  • json.c and json.h

  • nvme-models.c and nvme-models.h

  • plugin.c and plugin.h

and so on.


how to make VU command In NVMe-cli tool

First, NVMe-cli ver 0.7

you have to change files below

At first, I look at files on NVMe-cli ver 0.7

As you could see before, file of NVMe-clie ver 0.7 is the following :

$ cd nvme-cli--0.7
$ ls -v
CONTRIBUTING.md  NVME-VERSION-GEN  linux            nvme-ioctl.c  regress
Documentation    README.md         nvme.c           nvme-ioctl.h  scripts
LICENSE          completions       nvme.control.in  nvme-print.c  src
Makefile         debian            nvme.spec.in     nvme-print.h
$ cd linux 
$ ls -v
nvme.h

I changed files for VU command like this :

In linux directory.

  • nvme.h : what I change is because I have to add opcode(appointment between host and firmware),

         In other words, firmware perceives what kind of command host sent to firmware through opcode.
    

In nvme-cli–0.7 directory

  • nvme.c : this file includes the main functions. So I have to chage this file to add VU command with

  • nvme-print.c and nvme-print.h : this file includes how to print with return value from firmware,

  • nvme-ioctl.c and nvme-ioctl.h : basic way to use IOCTL in kernel is included in this file. So You have to change this file to use the suitable IOCTL to fit firmware

let’s make a example with virtual VU command.

Each files of NVMe-cli ver 0.7 that I mentioned.

if There is a functions for VU command, that function is used to have converstion with firmware to get information to make from Nand.

And with the function, Vendors determine each commands have different opcodes for conversation with firmware

let’s suppose the function name for VU command is __descriptor__ and the name of a VU command is __get-capability__

/nvme-cli-0.7/linux/nvme.h

In my case, I decided I should add opcodes for function and command into this file.

First opcode of Admin commands

// In /nvme-cli-0.7.linux/nvme.h

367 enum nvme_admin_opcode {
.......
384         nvme_admin_security_recv        = 0x82,
385         //function opcode of VU command  
386         nvme_admin_descriptor      = 0x00,
391 };

// AND
// The following structure I made for opcode of VU command

556 enum vu_opcode {
557         // OPCODE of VU command 
564         VU_OPCODE_GET_CAPABILITY                 = 0x01, // get capability
.......
569 };

As you can see the above code of an example. I made opcode in /nvme-cli-0.7/linux/nvme.h to manage it easily.

/nvme-cli-0.7/nvme-print.h and /nvme-cli-0.7/nvme-print.c

I just wanted to maintain the original state of nvme-cli-0.7. In order to keep structure of opensource(nvme-cli-0.7), I analyzed the nvme-cli-0.7.

And then, I realized all print function was in this files.

So, I used this files for print functions of VU commands.

let’s see an example code

you can see how to handle the return value from Nand, after issuing VU command.

// In /nvme-cli-0.7/nvme-print.h
// this is just changed for header file of nvme.c
// So, I just declare the function. 
11 
12 #ifndef COMMON_H
13 #define COMMON_H
14 
15 #include "linux/nvme.h"
16 
17 enum {
18         TERSE = 0x1u,   // only show a few useful fields
19         HUMAN = 0x2u,   // interpret some values for humans
20         VS    = 0x4u,   // print vendor specific data area
21         RAW   = 0x8u,   // just dump raw bytes
22 };
23
.......
33 void show_smart_log(struct nvme_smart_log *smart, unsigned int nsid, const char *devname);
34 void show_fw_log(struct nvme_firmware_log_page *fw_log, const char *devname);
.......
41 /* print function for VU commad */
42 
43 void nvme_show_capability(__u32 *vu_desc);
.......
45 #endif
// In /nvme-cli-0.7/nvme-print.c
// you can see the whole function of print.

1121 // output of get_capability opcode
1122 void nvme_show_capability(__u32 *vu_desc)
1123 {
1124         
1125                 printf("result                                  : %u\n", vu_desc[0]);
1126                 printf("num channels                            : %u\n", vu_desc[1]);  
.......
1147 }
1148 
/nvme-cli-0.7/nvme-ioctl.h and /nvme-cli-0.7/nvme-ioctl.c

these files include the main functions of ioctl

let’ see how to add ioctl for VU command

// In /nvme-cli-0.7/nvme-ioctl.c
// you can see what is header file of nvme-ioctl.c
12 #ifndef _NVME_LIB_H
13 #define _NVME_LIB_H
14 
15 #include <stdbool.h>
16 #include "linux/nvme.h"
17 
18 int nvme_get_nsid(int fd);
19 
20 /* Generic passthrough */
21 int nvme_submit_passthru(int fd, int ioctl_cmd, struct nvme_passthru_cmd *cmd);
22 
.......
29 
30 /* NVME_SUBMIT_IO */
31 int nvme_io(int fd, __u8 opcode, __u64 slba, __u16 nblocks, __u16 control,
32               __u32 dsmgmt, __u32 reftag, __u16 apptag,
33               __u16 appmask, void *data, void *metadata);
34 
.......
128 /* ioctl function for VU commad */
129 
130 int descriptor(int fd, __u32 cdw10, __u32 data_len, void *data);
131 int call_descriptor(int fd, __u32 len, void *data);
132 
133 #endif                          /* _NVME_LIB_H */

As you can see the above code, I made two functions, that is why it is easy to manange functions.

In other words, after extracting the common operation. I split multiple operation in one function into a operation in one function.

Here, I used opcode of ioctl function of VU command.

// In /nvme-cli-0.7/nvme-ioctl.c
// ioctl function of VU command
// depending on namespace, you can issue the VU command 
595 int descriptor(int fd,__u32 nsid __u32 cdw10, __u32 data_len, void *data)
596 {
597         struct nvme_admin_cmd cmd = {
598                 .opcode         = nvme_admin_descriptor(0x00),
599                 .nsid           = nsid,
600                 .cdw10          = cdw10,
.......
616                 .addr           = (__u64)(uintptr_t) data,
617                 .data_len       = data_len,
618         };
619 
620         return nvme_submit_admin_passthru(fd, &cmd);
621 }
622 
623 int call_descriptor(int fd,__u32 nsid, __u32 cdw10, __u32 len, void *data)
624 {
625         return send_descriptor(fd, nsid, cdw10, len, data);
626 }
627 

As you can see the above cmd of nvme_admin_cmd. cmd is different depending on vendors.

because vendors can decide cmd property to fit vendor’s spec.

The above spec bases in SMART-LOG command.

So detail for the admin_cmd of VU command depends on Vendor.

/nvme-cli-0.7/nvme.c

this file is very important. because when typing “./nvme –help”.

the upcoming result is from this file.

In /nvme-cli-0.7 
$ ./nvme --help
nvme-0.7
usage: nvme <command> [<device>] [<args>]

The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).

The following are all implemented sub-commands:
  list           	 List all NVMe devices and namespaces on machine
  id-ctrl        	 Send NVMe Identify Controller
.......
  get-capability 	 Show you capability
  move-vblk      	 Move pages from a source vblk to a destination vblk

See 'nvme help <command>' for more information on a specific command.

As you can see the above result of “./nvme –help”, you can find out get-capability sub-command.

This command, get-capability is function for VU command.

In /nvme-cli-0.7/nvme.c

78 #define COMMAND_LIST \
79         ENTRY(LIST, "list", "List all NVMe devices and namespaces on machine", list) \
80         ENTRY(ID_CTRL, "id-ctrl", "Send NVMe Identify Controller", id_ctrl) \
.......
120        ENTRY(GET_CAPABILITY, "get-capability", "Show you capability", get_capability) \
.......
127 #define ENTRY(i, n, h, f) \

In here, you can add functions and VU commands for “./nvme

In my case, I added get-capability of VU command.

120        ENTRY(GET_CAPABILITY, "get-capability", "Show you capability", get_capability) \

let’s see the actual function of get_capability in /nvme-cli-0.7/nvme.c

2793 // VU_OPCODE_GET_FLASH_CAP - Get Flash Capbility
2794 static int get_flash_capability(int argc, char **argv)
2795 {
2796         // this is for description per command. 
2797         const char *desc = "Get Initial capability to manage";
2798         const char *shell_test = "this is for shell script test";
2799         
2800         int err;
2801         __u32 *vu_desc = NULL; // VU command 
2802         void * buf = NULL; // buffer of VU command
2803          
.......
2805         struct config {
2806                 __u32 vu_desc_len;
2807                 __u32 shell_test;
2808         };
2809 
2810         struct config cfg = {
2811                 .vu_desc_len = 0,
2812                 .shell_test = 0,
2813         };
2814 
2815         const struct argconfig_commandline_options command_line_options[] = {
........          // for command line, "./nvme get-capability --help"
2817                 {"shell_test", 'e', "NUM", CFG_POSITIVE, &cfg.shell_test, no_argument, shell_test},
2818                 {0}
2819         };
2820         // this function parse arg of command, "./nvme get-capability"
2821         parse_and_open(argc, argv, desc, command_line_options, &cfg, sizeof(cfg));
2822 
2823         // getpagesize() = 4096 
2824         if (cfg.vu_desc_len == 0) {
2825                 cfg.vu_desc_len = 512 or 4096; // depending on vendor
2826                 if (posix_memalign(&buf, getpagesize(), cfg.vu_desc_len))
2827                         exit(ENOMEM);
2828                 memset(buf, 0, cfg.vu_desc_len);
2829         }
2830 
2831         vu_desc =(__u32 *)buf;
2832 
2833         vu_desc[0] = VU_OPCODE_GET_CAPABILITY(0x01);
2834 
2835         // this part pointer is tranlated to    
2836         err = call_descriptor(fd, cfg.vu_desc_len, buf);
2837         // if the VU command is excuted correctly.
2838         if(!err)
2839                 nvme_show_capability(buf);
2840         else {
.......
2844                 fprintf(stderr, "err number(hexa, deci(__u32))  : 0x%X, %u\n", err, err);
2845                 fprintf(stderr, "result(hexa, deci(__u32))      : 0x%X, %u\n", vu_desc[0], vu_desc[0]);
2846                 printf("get-capatilibty:Error\n");
2847         }
2848 
2849         if(buf)
2850                 free(buf);
2851 
2852         return err;
2853 }

As you can see the above example code, the above is basic format for VU command.

let’s see VU command with –help options

$ ./nvme get-capability --help
Usage: nvme get-capability <device> [OPTIONS]

Get Initial flash capability to manage

Options:
  [  --shell_test, -e ]                 --- this is for shell script test

So far, How to fix nvme-cli-0.7 for VU command.


Next, I will try to change nvme-cli-1.1 for VU command.

Second, NVMe-cli ver 1.1

let’s see nvme-cli-1.1 release’s file structure.

$ cd nvme-cli-1.1
$ ls -v
CONTRIBUTING.md  NVME-VERSION-GEN  cmd.h          debian        intel-nvme.c  linux            memblaze-nvme.h  nvme.spec.in    nvme-lightnvm.c  nvme-print.c  plugin.c  suffix.c
Documentation    README.md         cmd_handler.h  define_cmd.h  intel-nvme.h  lnvm-nvme.c      nvme.c           nvme-builtin.h  nvme-lightnvm.h  nvme-print.h  plugin.h  suffix.h
LICENSE          argconfig.c       common.h       fabrics.c     json.c        lnvm-nvme.h      nvme.control.in  nvme-ioctl.c    nvme-models.c    parser.c      regress   tests
Makefile         argconfig.h       completions    fabrics.h     json.h        memblaze-nvme.c  nvme.h           nvme-ioctl.h    nvme-models.h    parser.h      scripts
$ cd linux
$ ls -v
lightnvm.h  nvme.h  nvme_ioctl.h

As you can see the above result, many things is different from nvme-cli-0.7 release.

BUT, If you want to make VU command in nvme-cli-1.1, it is easy under a similar way.

In other words, Basically, when you change the nvme-cli-1.1 with a similar way to nvme-cli-0.7 without plugin mode, files that you have to change is similar to nvme-cli-0.7

Without plugin mode, that is temporary way.

first, I will explain to you how to change nvme-cli-1.1 for VU command without plugin mode.

Without writting opcode in /nvme-cli-1.1/linux/nvme.h, everything is almost the same.

The files that I have to change :

  • nvme.c : this is the same from nvme-cli-0.7, this file includes function for the VU command

  • nvme-print.c and nvme-print.h : this includes function of print for VU command as well.

  • nvme-ioctl.c and nvme-ioctl.h : this includes funtion of ioctl for VU command as well.

  • nvme-builtin.h : this file is new, just on ./nvme , this is file for

Each files of NVMe-cli ver 1.1

/nvme-cli-1.1/nvme-builtin.h

In my case, so as to register VU command, I had to change this file first.

let’s see the code.

$ vim nvme-builtin.h

// In /nvme-cli-1.1/nvme-builtin.h
// in order to register the name of command and function. 

12 #undef CMD_INC_FILE
13 #define CMD_INC_FILE nvme-builtin
14 
15 #if !defined(NVME_BUILTIN) || defined(CMD_HEADER_MULTI_READ)
16 #define NVME_BUILTIN
17 
18 #include "cmd.h"
19 
20 COMMAND_LIST(
21         ENTRY("list", "List all NVMe devices and namespaces on machine", list)
22         ENTRY("id-ctrl", "Send NVMe Identify Controller", id_ctrl)
23         ENTRY("id-ns", "Send NVMe Identify Namespace, display structure", id_ns)
24         ENTRY("list-ns", "Send NVMe Identify List, display structure", list_ns)
25         ENTRY("create-ns", "Creates a namespace with the provided parameters", create_ns)
26         ENTRY("delete-ns", "Deletes a namespace from the controller", delete_ns)
27         ENTRY("attach-ns", "Attaches a namespace to requested controller(s)", attach_ns)
28         ENTRY("detach-ns", "Detaches a namespace from requested controller(s)", detach_ns)
29         ENTRY("list-ctrl", "Send NVMe Identify Controller List, display structure", list_ctrl)
30         ENTRY("get-ns-id", "Retrieve the namespace ID of opened block device", get_ns_id)
31         ENTRY("get-log", "Generic NVMe get log, returns log in raw format", get_log)
32         ENTRY("fw-log", "Retrieve FW Log, show it", get_fw_log)
33         ENTRY("smart-log", "Retrieve SMART Log, show it", get_smart_log)
34         ENTRY("error-log", "Retrieve Error Log, show it", get_error_log)
35         ENTRY("get-feature", "Get feature and show the resulting value", get_feature)
36         ENTRY("set-feature", "Set a feature and show the resulting value", set_feature)
37         ENTRY("format", "Format namespace with new block format", format)
38         ENTRY("fw-activate", "Activate new firmware slot", fw_activate)
39         ENTRY("fw-download", "Download new firmware", fw_download)
40         ENTRY("admin-passthru", "Submit arbitrary admin command, return results", admin_passthru)
41         ENTRY("io-passthru", "Submit an arbitrary IO command, return results", io_passthru)
42         ENTRY("security-send", "Submit a Security Send command, return results", sec_send)
43         ENTRY("security-recv", "Submit a Security Receive command, return results", sec_recv)
44         ENTRY("resv-acquire", "Submit a Reservation Acquire, return results", resv_acquire)
45         ENTRY("resv-register", "Submit a Reservation Register, return results", resv_register)
46         ENTRY("resv-release", "Submit a Reservation Release, return results", resv_release)
47         ENTRY("resv-report", "Submit a Reservation Report, return results", resv_report)
48         ENTRY("dsm", "Submit a Data Set Management command, return results", dsm)
49         ENTRY("flush", "Submit a Flush command, return results", flush)
50         ENTRY("compare", "Submit a Compare command, return results", compare)
51         ENTRY("read", "Submit a read command, return results", read_cmd)
52         ENTRY("write", "Submit a write command, return results", write_cmd)
53         ENTRY("write-zeroes", "Submit a write zeroes command, return results", write_zeroes)
54         ENTRY("write-uncor", "Submit a write uncorrectable command, return results", write_uncor)
55         ENTRY("reset", "Resets the controller", reset)
56         ENTRY("subsystem-reset", "Resets the controller", subsystem_reset)
57         ENTRY("show-regs", "Shows the controller registers. Requires admin character device", show_registers)
58         ENTRY("discover", "Discover NVMeoF subsystems", discover_cmd)
59         ENTRY("connect-all", "Discover and Connect to NVMeoF subsystems", connect_all_cmd)
60         ENTRY("connect", "Connect to NVMeoF subsystem", connect_cmd)
61         ENTRY("disconnect", "Disconnect from NVMeoF subsystem", disconnect_cmd)
62         // For VU command 
63         ENTRY("get-information", "Get information from NVMe subsystem",get_information)
64 );
65 
66 #endif
67 
68 #include "define_cmd.h"

As you can see the above code, You can check the above contents with “./nvme –help”

$ ./nvme --help
nvme-1.1
usage: nvme <command> [<device>] [<args>]

The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).

The following are all implemented sub-commands:
  list            List all NVMe devices and namespaces on machine
  id-ctrl         Send NVMe Identify Controller
  id-ns           Send NVMe Identify Namespace, display structure
  list-ns         Send NVMe Identify List, display structure
  create-ns       Creates a namespace with the provided parameters
  delete-ns       Deletes a namespace from the controller
  attach-ns       Attaches a namespace to requested controller(s)
  detach-ns       Detaches a namespace from requested controller(s)
  list-ctrl       Send NVMe Identify Controller List, display structure
  get-ns-id       Retrieve the namespace ID of opened block device
  get-log         Generic NVMe get log, returns log in raw format
  fw-log          Retrieve FW Log, show it
  smart-log       Retrieve SMART Log, show it
  error-log       Retrieve Error Log, show it
  get-feature     Get feature and show the resulting value
  set-feature     Set a feature and show the resulting value
  format          Format namespace with new block format
  fw-activate     Activate new firmware slot
  fw-download     Download new firmware
  admin-passthru  Submit arbitrary admin command, return results
  io-passthru     Submit an arbitrary IO command, return results
  security-send   Submit a Security Send command, return results
  security-recv   Submit a Security Receive command, return results
  resv-acquire    Submit a Reservation Acquire, return results
  resv-register   Submit a Reservation Register, return results
  resv-release    Submit a Reservation Release, return results
  resv-report     Submit a Reservation Report, return results
  dsm             Submit a Data Set Management command, return results
  flush           Submit a Flush command, return results
  compare         Submit a Compare command, return results
  read            Submit a read command, return results
  write           Submit a write command, return results
  write-zeroes    Submit a write zeroes command, return results
  write-uncor     Submit a write uncorrectable command, return results
  reset           Resets the controller
  subsystem-reset Resets the controller
  show-regs       Shows the controller registers. Requires admin character device
  discover        Discover NVMeoF subsystems
  connect-all     Discover and Connect to NVMeoF subsystems
  connect         Connect to NVMeoF subsystem
  disconnect      Disconnect from NVMeoF subsystem
  get-information Get information from NVMe subsystem
  version         Shows the program version
  help            Display this help
.......

you could see new command, get-information, additionally.

/nvme-cli-1.1/nvme-print.c and /nvme-cli-1.1/nvme-print.h

In my case, I added function of print for get-information of VU command.

First of all, let’s see the /nvme-cli-1.1/nvme-print.h

// In /nvme-cli-1.1/nvme-print.h
13 #ifndef NVME_PRINT_H
14 #define NVME_PRINT_H
15 
16 #include "nvme.h"
17 #include <inttypes.h>
18 
19 enum {
20         TERSE = 0x1u,   // only show a few useful fields
21         HUMAN = 0x2u,   // interpret some values for humans
22         VS    = 0x4u,   // print vendor specific data area
23         RAW   = 0x8u,   // just dump raw bytes
24 };
25 
26 void d(unsigned char *buf, int len, int width, int group);
27 void d_raw(unsigned char *buf, unsigned len);
28 
29 uint64_t int48_to_long(__u8 *data);
30 
31 void __show_nvme_id_ctrl(struct nvme_id_ctrl *ctrl, unsigned int mode, void (*vendor_show)(__u8 *vs));
32 void show_nvme_id_ctrl(struct nvme_id_ctrl *ctrl, unsigned int mode);
.......
57 //For VU command 
58 void show_information(__u32 *result, unsigned int nsid, const char *devname);
59 
60 
61 #endif

this file has role as header file.

In /nvme-cli-1.1/nvme-print.c
1791 // For VU command of SK hynix SSD
1792 void show_information(__u32 *result, unsigned int nsid, const char *devname)
1793 {
1794         printf("information NVME device:%s namespace-id:%x\n", devname, nsid);
1795         printf("dw[0] : %u, 0x%x\n", result[0], result[0]);
.........
1802         printf("dw[3] : %u, 0x%x\n", result[7], result[7]);
1803 }
1804 

on the above function, with buffer between host and firmware.

you can print the return value.

/nvme-cli-1.1/nvme-ioctl.c and /nvme-cli-1.1/nvme-ioctl.h

this is also the same from nvme-cli-0.7’s nvme-ioctol file.

In /nvme-cli-1.1/nvme-ioctl.h
11 #ifndef _NVME_LIB_H
12 #define _NVME_LIB_H
13 
14 #include <linux/types.h>
15 #include <stdbool.h>
16 #include "linux/nvme_ioctl.h"
17 #include "nvme.h"
18 
19 int nvme_get_nsid(int fd);
20 
21 /* Generic passthrough */
22 int nvme_submit_passthru(int fd, int ioctl_cmd, struct nvme_passthru_cmd *cmd);
23 
24 int nvme_passthru(int fd, int ioctl_cmd, __u8 opcode, __u8 flags,
25                   __u16 rsvd, __u32 nsid, __u32 cdw2, __u32 cdw3,
26                   __u32 cdw10, __u32 cdw11, __u32 cdw12,
27                   __u32 cdw13, __u32 cdw14, __u32 cdw15,
28                   __u32 data_len, void *data, __u32 metadata_len,
29                   void *metadata, __u32 timeout_ms, __u32 *result);
30 
31 /* NVME_SUBMIT_IO */
32 int nvme_io(int fd, __u8 opcode, __u64 slba, __u16 nblocks, __u16 control,
33               __u32 dsmgmt, __u32 reftag, __u16 apptag,
34               __u16 appmask, void *data, void *metadata);
.........
127 int nvme_subsystem_reset(int fd);
128 int nvme_reset_controller(int fd);
129 /*For VU command*/
130 
131 int descriptor(int fd, __u32 nsid, __u32 cdw10, __u32 data_len, void *data);
132 int nvme_information(int fd, __u32 nsid, __u32 len, void *data);
133 
134 
135 #endif                          /* _NVME_LIB_H */

This file is also similar to /nvme-cli-0.7/nvme-ioctl.h

it’s just different from function name.

In /nvme-cli-1.1/nvme-ioctl.c

605 // For VU command
606 
607 int descriptor(int fd, __u32 nsid, __u32 cdw10, __u32 data_len, void *data)
608 {
609   struct nvme_admin_cmd cmd = {
610         .opcode         = 0x00,//nvme_admin_descriptor;
611         .nsid           = nsid, 
612         .cdw10          = cdw10,
613         .addr           = (__u64)(uintptr_t) data,
614         .data_len       = data_len,
615   };
616 
617         return nvme_submit_admin_passthru(fd, &cmd);
618 }
619 
620 
621 int nvme_information(int fd, __u32 nsid, __u32 len, void *data)
622 {
623   return descriptor(fd, nsid, len, len, data);
624 }

The above example code is also based on smart-log of nvme-cli-1.1.

let’s last see VU command function.

/nvme-cli-1.1/nvme.c

In this file, I used get_smart_log function of get_inforamtion that is virtual VU command.

184 // FOR VU command
185 // get_information
186 static int get_information(int argc, char **argv, struct command *cmd, struct plugin *plugin)
187 {
188         const char *desc = "Retrieve information for the given device "\
189                         "(or optionally a namespace) in either decoded format "\
190                         "(default) or binary.";
191         const char *namespace = "(optional) desired namespace";
192         const char *raw = "output in binary format";
193         int err, fmt, fd;
194         __u32 *vu_desc = NULL; // VU command 
195         void *buf = NULL; // Buffer for VU command
196 
197         struct config {
198                 __u32 vu_desc_len;
199                 __u32 namespace_id;
200                 int   raw_binary;
201                 char *output_format;
202         };
203 
204         struct config cfg = {
205                 .vu_desc_len = 0,
206                 .namespace_id = 0xffffffff,
207                 .output_format = "normal",
208         };
209 
210         const struct argconfig_commandline_options command_line_options[] = {
211                 {"namespace-id",  'n', "NUM", CFG_POSITIVE, &cfg.namespace_id,  required_argument, namespace},
212                 {"output-format", 'o', "FMT", CFG_STRING,   &cfg.output_format, required_argument, output_format },
213                 {"raw-binary",    'b', "",    CFG_NONE,     &cfg.raw_binary,    no_argument,       raw},
214                 {NULL}
215         };
216 
217         fd = parse_and_open(argc, argv, desc, command_line_options, &cfg, sizeof(cfg));
218         if (fd < 0)
219                 return fd;
220          
221         fmt = validate_output_format(cfg.output_format);
222         if (fmt < 0)
223                 return fmt;
224         if (cfg.raw_binary)
225                 fmt = BINARY;
226 
227         // getpagesize() = 4096
228         if (cfg.vu_desc_len == 0){
229             cfg.vu_desc_len = 4096 or 512; depending on vendors
230             if (posix_memalign(&buf, getpagesize(), cfg.vu_desc_len))
231                 exit(ENOMEM);
232             memset(buf, 0, cfg.vu_desc_len);
233         }
234 
235         vu_desc = (__u32 *)buf;
236         vu_desc[0] = 0x01; //VU_OPCODE_GET_INFORMATION;
237 
238 
239         err = nvme_information(fd, cfg.namespace_id, cfg.vu_desc_len, buf);
240         if (!err) {
241                 if (fmt == BINARY)
242                         fprintf(stderr, "this does not support Binary mode\n");
243                 else if (fmt == JSON)
244                         fprintf(stderr, "this does not support json mode\n");
245                 else
246                         show_information(buf, cfg.namespace_id, devicename);
247         }
248         else if (err > 0)
249                 fprintf(stderr, "NVMe Status:%s(%x)\n",
250                                         nvme_status_to_string(err), err);
251         else
252                 perror("get information error");
253
254         if (buf != 0)
255           free(buf);
256
257         return err;
258 }

let’s see output of help option of get-information command

In /nvme-cli-1.1
$ ./nvme get-information --help
Usage: nvme get-information <device> [OPTIONS]

Retrieve information for the given device (or optionally a namespace) in
either decoded format (default) or binary.

Options:
  [  --namespace-id=<NUM>, -n <NUM> ]   --- (optional) desired namespace
  [  --output-format=<FMT>, -o <FMT> ]  --- Output format: normal|json|binary
  [  --raw-binary, -b ]                 --- output in binary format
  

Appendix A

At first, nvme-builtin.h

#undef CMD_INC_FILE 
#define CMD_INC_FILE nvme-builtin 
 
#if !defined(NVME_BUILTIN) || defined(CMD_HEADER_MULTI_READ) 
#define NVME_BUILTIN 
 
#include "cmd.h" 
 
COMMAND_LIST( 
        ENTRY("list", "List all NVMe devices and namespaces on machine", list) 
        ENTRY("id-ctrl", "Send NVMe Identify Controller", id_ctrl) 
        ENTRY("id-ns", "Send NVMe Identify Namespace, display structure", id_ns) 
        ENTRY("list-ns", "Send NVMe Identify List, display structure", list_ns) 
        ENTRY("create-ns", "Creates a namespace with the provided parameters", create_ns) 
        ENTRY("delete-ns", "Deletes a namespace from the controller", delete_ns) 
        ENTRY("attach-ns", "Attaches a namespace to requested controller(s)", attach_ns) 
        ENTRY("detach-ns", "Detaches a namespace from requested controller(s)", detach_ns) 
        ENTRY("list-ctrl", "Send NVMe Identify Controller List, display structure", list_ctrl) 
        ENTRY("get-ns-id", "Retrieve the namespace ID of opened block device", get_ns_id) 
        ENTRY("get-log", "Generic NVMe get log, returns log in raw format", get_log) 
        ENTRY("fw-log", "Retrieve FW Log, show it", get_fw_log) 
        ENTRY("smart-log", "Retrieve SMART Log, show it", get_smart_log) 
        ENTRY("error-log", "Retrieve Error Log, show it", get_error_log) 
        ENTRY("get-feature", "Get feature and show the resulting value", get_feature) 
        ENTRY("set-feature", "Set a feature and show the resulting value", set_feature) 
        ENTRY("format", "Format namespace with new block format", format) 
        ENTRY("fw-activate", "Activate new firmware slot", fw_activate) 
        ENTRY("fw-download", "Download new firmware", fw_download) 
        ENTRY("admin-passthru", "Submit arbitrary admin command, return results", admin_passthru) 
        ENTRY("io-passthru", "Submit an arbitrary IO command, return results", io_passthru) 
        ENTRY("security-send", "Submit a Security Send command, return results", sec_send) 
        ENTRY("security-recv", "Submit a Security Receive command, return results", sec_recv) 
        ENTRY("resv-acquire", "Submit a Reservation Acquire, return results", resv_acquire) 
        ENTRY("resv-register", "Submit a Reservation Register, return results", resv_register) 
        ENTRY("resv-release", "Submit a Reservation Release, return results", resv_release) 
        ENTRY("resv-report", "Submit a Reservation Report, return results", resv_report) 
        ENTRY("dsm", "Submit a Data Set Management command, return results", dsm) 
        ENTRY("flush", "Submit a Flush command, return results", flush) 
        ENTRY("compare", "Submit a Compare command, return results", compare) 
        ENTRY("read", "Submit a read command, return results", read_cmd) 
        ENTRY("write", "Submit a write command, return results", write_cmd) 
        ENTRY("write-zeroes", "Submit a write zeroes command, return results", write_zeroes) 
        ENTRY("write-uncor", "Submit a write uncorrectable command, return results", write_uncor) 
        ENTRY("reset", "Resets the controller", reset) 
        ENTRY("subsystem-reset", "Resets the controller", subsystem_reset) 
        ENTRY("show-regs", "Shows the controller registers. Requires admin character device", show_registers) 
        ENTRY("discover", "Discover NVMeoF subsystems", discover_cmd) 
        ENTRY("connect-all", "Discover and Connect to NVMeoF subsystems", connect_all_cmd) 
        ENTRY("connect", "Connect to NVMeoF subsystem", connect_cmd) 
        ENTRY("disconnect", "Disconnect from NVMeoF subsystem", disconnect_cmd) 
); 
 
#endif 
 
#include "define_cmd.h"

Second, nvme.c

static int get_smart_log(int argc, char **argv, struct command *cmd, struct plugin *plugin)
{
        struct nvme_smart_log smart_log;
        const char *desc = "Retrieve SMART log for the given device "\
                        "(or optionally a namespace) in either decoded format "\
                        "(default) or binary.";
        const char *namespace = "(optional) desired namespace";
        const char *raw = "output in binary format";
        int err, fmt, fd;

        struct config {
                __u32 namespace_id;
                int   raw_binary;
                char *output_format;
        };

        struct config cfg = {
                .namespace_id = 0xffffffff,
                .output_format = "normal",
        };

        const struct argconfig_commandline_options command_line_options[] = {
                {"namespace-id",  'n', "NUM", CFG_POSITIVE, &cfg.namespace_id,  required_argument, namespace},
                {"output-format", 'o', "FMT", CFG_STRING,   &cfg.output_format, required_argument, output_format },
                {"raw-binary",    'b', "",    CFG_NONE,     &cfg.raw_binary,    no_argument,       raw},
                {NULL}
        };

        fd = parse_and_open(argc, argv, desc, command_line_options, &cfg, sizeof(cfg));
        if (fd < 0)
                return fd;

        fmt = validate_output_format(cfg.output_format);
        if (fmt < 0)
                return fmt;
        if (cfg.raw_binary)
                fmt = BINARY;

        err = nvme_smart_log(fd, cfg.namespace_id, &smart_log);
        if (!err) {
                if (fmt == BINARY)
                        d_raw((unsigned char *)&smart_log, sizeof(smart_log));
                else if (fmt == JSON)
                        json_smart_log(&smart_log, cfg.namespace_id, devicename);
                else
                        show_smart_log(&smart_log, cfg.namespace_id, devicename);
        }
        else if (err > 0)
                fprintf(stderr, "NVMe Status:%s(%x)\n",
                                        nvme_status_to_string(err), err);
        else
                perror("smart log");
        return err;
}

From now on, just function to print

Third, nvme-print.h

#ifndef NVME_PRINT_H
#define NVME_PRINT_H

#include "nvme.h"
#include <inttypes.h>

enum {
        TERSE = 0x1u,   // only show a few useful fields
        HUMAN = 0x2u,   // interpret some values for humans
        VS    = 0x4u,   // print vendor specific data area
        RAW   = 0x8u,   // just dump raw bytes
};

void d(unsigned char *buf, int len, int width, int group);
void d_raw(unsigned char *buf, unsigned len);

uint64_t int48_to_long(__u8 *data);

void __show_nvme_id_ctrl(struct nvme_id_ctrl *ctrl, unsigned int mode, void (*vendor_show)(__u8 *vs));
void show_nvme_id_ctrl(struct nvme_id_ctrl *ctrl, unsigned int mode);
void show_nvme_id_ns(struct nvme_id_ns *ns, unsigned int flags);
void show_nvme_resv_report(struct nvme_reservation_status *status);
void show_lba_range(struct nvme_lba_range_type *lbrt, int nr_ranges);
void show_error_log(struct nvme_error_log_page *err_log, int entries, const char *devname);
void show_intel_smart_log(struct nvme_additional_smart_log *smart, unsigned int nsid, const char *devname);
void show_smart_log(struct nvme_smart_log *smart, unsigned int nsid, const char *devname);
void show_fw_log(struct nvme_firmware_log_page *fw_log, const char *devname);
void show_ctrl_registers(void *bar, unsigned int mode);

void nvme_feature_show_fields(__u32 fid, unsigned int result, unsigned char *buf);
char *nvme_status_to_string(__u32 status);
char *nvme_select_to_string(int sel);
char *nvme_feature_to_string(int feature);

void json_nvme_id_ctrl(struct nvme_id_ctrl *ctrl, unsigned int mode);
void json_nvme_id_ns(struct nvme_id_ns *ns, unsigned int flags);
void json_nvme_resv_report(struct nvme_reservation_status *status);
void json_error_log(struct nvme_error_log_page *err_log, int entries, const char *devname);
void json_smart_log(struct nvme_smart_log *smart, unsigned int nsid, const char *devname);
void json_add_smart_log(struct nvme_additional_smart_log *smart,
                        unsigned int nsid, const char *devname);
void json_fw_log(struct nvme_firmware_log_page *fw_log, const char *devname);
void json_print_list_items(struct list_item *items, unsigned amnt);


#endif

Addtionally, nvme-print.c file

finally, IF you change this file. you can make nvme-cli tool to support VU command.

be careful, what I am saying is what version of NVMe-cli, because later on, version will be changed.

BUT, If you want to support man operation of linux. you need a little more

let’s make a code for VU command

Ahead of making VU command, let’s check what version of nvme-cli is right after “git clone https://github.com/linux-nvme/nvme-cli.git”

in nvme directory, type ./nvme after that, you would get the following result.

$ ./nvme
nvme-1.1.2.g0142
usage: nvme <command> [<device>] [<args>]

The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).

The following are all implemented sub-commands:
  list            List all NVMe devices and namespaces on machine
  id-ctrl         Send NVMe Identify Controller
  id-ns           Send NVMe Identify Namespace, display structure
  list-ns         Send NVMe Identify List, display structure
  create-ns       Creates a namespace with the provided parameters
  delete-ns       Deletes a namespace from the controller
  attach-ns       Attaches a namespace to requested controller(s)
  detach-ns       Detaches a namespace from requested controller(s)
  list-ctrl       Send NVMe Identify Controller List, display structure
  get-ns-id       Retrieve the namespace ID of opened block device
  get-log         Generic NVMe get log, returns log in raw format
  fw-log          Retrieve FW Log, show it
  smart-log       Retrieve SMART Log, show it
  error-log       Retrieve Error Log, show it
  get-feature     Get feature and show the resulting value
  set-feature     Set a feature and show the resulting value
  format          Format namespace with new block format
  fw-activate     Activate new firmware slot
  fw-download     Download new firmware
  admin-passthru  Submit arbitrary admin command, return results
  io-passthru     Submit an arbitrary IO command, return results
  security-send   Submit a Security Send command, return results
  security-recv   Submit a Security Receive command, return results
  resv-acquire    Submit a Reservation Acquire, return results
  resv-register   Submit a Reservation Register, return results
  resv-release    Submit a Reservation Release, return results
  resv-report     Submit a Reservation Report, return results
  dsm             Submit a Data Set Management command, return results
  flush           Submit a Flush command, return results
  compare         Submit a Compare command, return results
  read            Submit a read command, return results
  write           Submit a write command, return results
  write-zeroes    Submit a write zeroes command, return results
  write-uncor     Submit a write uncorrectable command, return results
  reset           Resets the controller
  subsystem-reset Resets the controller
  show-regs       Shows the controller registers. Requires admin character device
  discover        Discover NVMeoF subsystems
  connect-all     Discover and Connect to NVMeoF subsystems
  connect         Connect to NVMeoF subsystem
  disconnect      Disconnect from NVMeoF subsystem
  version         Shows the program version
  help            Display this help

See 'nvme help <command>' for more information on a specific command

The following are all installed plugin extensions:
  intel           Intel vendor specific extensions
  lnvm            LightNVM specific extensions
  memblaze        Memblaze vendor specific extensions

See 'nvme <plugin> help' for more information on a plugin

As you can see below, the version of NVMe-cli is nvme-1.1.2.g0142.

So I could determine What version I would make VU command on.

I decided That version is 1.1, you can get download in release of the officail site of nvme-cli.

here is v1.1 of nvme-cli release.

let’s start making VU command on NVMe-cli 1.1 release

First, You have to download nvme-cli release version 1.1.

In my case, I downloaded nvme-cli-1.1.zip

let’s uncompress that file.

$ unzip nvme-cli-1.1.zip

After that, as you can see the list with ls command, you would find out nvme-cli-1.1

$ ls
nvme-cli-0.7  nvme-cli-0.7.tar  nvme-cli-1.1

And then you need to chang some files to insert VU command into opensource(nvme-cli)

as you read the above files,first, I will change the file, nvme-builtin.h.

nvme-builtin.h file

above all, I would exlain to you about code of nvme-builtin.h

#undef CMD_INC_FILE
#define CMD_INC_FILE nvme-builtin

#if !defined(NVME_BUILTIN) || defined(CMD_HEADER_MULTI_READ)
#define NVME_BUILTIN

#include "cmd.h"

COMMAND_LIST(
        ENTRY("list", "List all NVMe devices and namespaces on machine", list)
        ......
);

#endif

#include "define_cmd.h"

As you see the above code. ENTRY indicates the list of ./nvme command.

“list” - command name

“List all NVMe device and namespaces on machine” - summary about what role of this command is.

list - function name in nvme.c file

you can check up what I am saying to you indirectly. type ./nvme in nvme-cli directory.

$ ./nvme
nvme-1.1.2.g0142
usage: nvme <command> [<device>] [<args>]

The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).

The following are all implemented sub-commands:
  list            List all NVMe devices and namespaces on machine
  ....
  
See 'nvme help <command>' for more information on a specific command

The following are all installed plugin extensions:
  intel           Intel vendor specific extensions
  lnvm            LightNVM specific extensions
  memblaze        Memblaze vendor specific extensions

So you change this file like this


```c
#undef CMD_INC_FILE
#define CMD_INC_FILE nvme-builtin

#if !defined(NVME_BUILTIN) || defined(CMD_HEADER_MULTI_READ)
#define NVME_BUILTIN

#include "cmd.h"

COMMAND_LIST(
        ENTRY("list", "List all NVMe devices and namespaces on machine", list)
        ......
        // add ENTRY of VU command In here
        ENTRY("command of nvme-clie for VU command", "Description of NVMe-cli command", "Function name of nvme.c file for this Command")
        // For example
        ENTRY("nvme-waf", "get WAF from NVMe subsystem", get_waf)
);

#endif

#include "define_cmd.h"
.......

nvme.c and nvme.h

–>