1 package intelrdt 2 3 import ( 4 "bytes" 5 "errors" 6 "fmt" 7 "os" 8 "path/filepath" 9 "strconv" 10 "strings" 11 "sync" 12 13 "github.com/moby/sys/mountinfo" 14 "golang.org/x/sys/unix" 15 16 "github.com/opencontainers/runc/libcontainer/cgroups/fscommon" 17 "github.com/opencontainers/runc/libcontainer/configs" 18 ) 19 20 /* 21 * About Intel RDT features: 22 * Intel platforms with new Xeon CPU support Resource Director Technology (RDT). 23 * Cache Allocation Technology (CAT) and Memory Bandwidth Allocation (MBA) are 24 * two sub-features of RDT. 25 * 26 * Cache Allocation Technology (CAT) provides a way for the software to restrict 27 * cache allocation to a defined 'subset' of L3 cache which may be overlapping 28 * with other 'subsets'. The different subsets are identified by class of 29 * service (CLOS) and each CLOS has a capacity bitmask (CBM). 30 * 31 * Memory Bandwidth Allocation (MBA) provides indirect and approximate throttle 32 * over memory bandwidth for the software. A user controls the resource by 33 * indicating the percentage of maximum memory bandwidth or memory bandwidth 34 * limit in MBps unit if MBA Software Controller is enabled. 35 * 36 * More details about Intel RDT CAT and MBA can be found in the section 17.18 37 * of Intel Software Developer Manual: 38 * https://software.intel.com/en-us/articles/intel-sdm 39 * 40 * About Intel RDT kernel interface: 41 * In Linux 4.10 kernel or newer, the interface is defined and exposed via 42 * "resource control" filesystem, which is a "cgroup-like" interface. 43 * 44 * Comparing with cgroups, it has similar process management lifecycle and 45 * interfaces in a container. But unlike cgroups' hierarchy, it has single level 46 * filesystem layout. 47 * 48 * CAT and MBA features are introduced in Linux 4.10 and 4.12 kernel via 49 * "resource control" filesystem. 50 * 51 * Intel RDT "resource control" filesystem hierarchy: 52 * mount -t resctrl resctrl /sys/fs/resctrl 53 * tree /sys/fs/resctrl 54 * /sys/fs/resctrl/ 55 * |-- info 56 * | |-- L3 57 * | | |-- cbm_mask 58 * | | |-- min_cbm_bits 59 * | | |-- num_closids 60 * | |-- L3_MON 61 * | | |-- max_threshold_occupancy 62 * | | |-- mon_features 63 * | | |-- num_rmids 64 * | |-- MB 65 * | |-- bandwidth_gran 66 * | |-- delay_linear 67 * | |-- min_bandwidth 68 * | |-- num_closids 69 * |-- ... 70 * |-- schemata 71 * |-- tasks 72 * |-- <clos> 73 * |-- ... 74 * |-- schemata 75 * |-- tasks 76 * 77 * For runc, we can make use of `tasks` and `schemata` configuration for L3 78 * cache and memory bandwidth resources constraints. 79 * 80 * The file `tasks` has a list of tasks that belongs to this group (e.g., 81 * <container_id>" group). Tasks can be added to a group by writing the task ID 82 * to the "tasks" file (which will automatically remove them from the previous 83 * group to which they belonged). New tasks created by fork(2) and clone(2) are 84 * added to the same group as their parent. 85 * 86 * The file `schemata` has a list of all the resources available to this group. 87 * Each resource (L3 cache, memory bandwidth) has its own line and format. 88 * 89 * L3 cache schema: 90 * It has allocation bitmasks/values for L3 cache on each socket, which 91 * contains L3 cache id and capacity bitmask (CBM). 92 * Format: "L3:<cache_id0>=<cbm0>;<cache_id1>=<cbm1>;..." 93 * For example, on a two-socket machine, the schema line could be "L3:0=ff;1=c0" 94 * which means L3 cache id 0's CBM is 0xff, and L3 cache id 1's CBM is 0xc0. 95 * 96 * The valid L3 cache CBM is a *contiguous bits set* and number of bits that can 97 * be set is less than the max bit. The max bits in the CBM is varied among 98 * supported Intel CPU models. Kernel will check if it is valid when writing. 99 * e.g., default value 0xfffff in root indicates the max bits of CBM is 20 100 * bits, which mapping to entire L3 cache capacity. Some valid CBM values to 101 * set in a group: 0xf, 0xf0, 0x3ff, 0x1f00 and etc. 102 * 103 * Memory bandwidth schema: 104 * It has allocation values for memory bandwidth on each socket, which contains 105 * L3 cache id and memory bandwidth. 106 * Format: "MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;..." 107 * For example, on a two-socket machine, the schema line could be "MB:0=20;1=70" 108 * 109 * The minimum bandwidth percentage value for each CPU model is predefined and 110 * can be looked up through "info/MB/min_bandwidth". The bandwidth granularity 111 * that is allocated is also dependent on the CPU model and can be looked up at 112 * "info/MB/bandwidth_gran". The available bandwidth control steps are: 113 * min_bw + N * bw_gran. Intermediate values are rounded to the next control 114 * step available on the hardware. 115 * 116 * If MBA Software Controller is enabled through mount option "-o mba_MBps": 117 * mount -t resctrl resctrl -o mba_MBps /sys/fs/resctrl 118 * We could specify memory bandwidth in "MBps" (Mega Bytes per second) unit 119 * instead of "percentages". The kernel underneath would use a software feedback 120 * mechanism or a "Software Controller" which reads the actual bandwidth using 121 * MBM counters and adjust the memory bandwidth percentages to ensure: 122 * "actual memory bandwidth < user specified memory bandwidth". 123 * 124 * For example, on a two-socket machine, the schema line could be 125 * "MB:0=5000;1=7000" which means 5000 MBps memory bandwidth limit on socket 0 126 * and 7000 MBps memory bandwidth limit on socket 1. 127 * 128 * For more information about Intel RDT kernel interface: 129 * https://www.kernel.org/doc/Documentation/x86/intel_rdt_ui.txt 130 * 131 * An example for runc: 132 * Consider a two-socket machine with two L3 caches where the default CBM is 133 * 0x7ff and the max CBM length is 11 bits, and minimum memory bandwidth of 10% 134 * with a memory bandwidth granularity of 10%. 135 * 136 * Tasks inside the container only have access to the "upper" 7/11 of L3 cache 137 * on socket 0 and the "lower" 5/11 L3 cache on socket 1, and may use a 138 * maximum memory bandwidth of 20% on socket 0 and 70% on socket 1. 139 * 140 * "linux": { 141 * "intelRdt": { 142 * "l3CacheSchema": "L3:0=7f0;1=1f", 143 * "memBwSchema": "MB:0=20;1=70" 144 * } 145 * } 146 */ 147 148 type Manager struct { 149 mu sync.Mutex 150 config *configs.Config 151 id string 152 path string 153 } 154 155 // NewManager returns a new instance of Manager, or nil if the Intel RDT 156 // functionality is not specified in the config, available from hardware or 157 // enabled in the kernel. 158 func NewManager(config *configs.Config, id string, path string) *Manager { 159 if config.IntelRdt == nil { 160 return nil 161 } 162 if _, err := Root(); err != nil { 163 // Intel RDT is not available. 164 return nil 165 } 166 return newManager(config, id, path) 167 } 168 169 // newManager is the same as NewManager, except it does not check if the feature 170 // is actually available. Used by unit tests that mock intelrdt paths. 171 func newManager(config *configs.Config, id string, path string) *Manager { 172 return &Manager{ 173 config: config, 174 id: id, 175 path: path, 176 } 177 } 178 179 const ( 180 intelRdtTasks = "tasks" 181 ) 182 183 var ( 184 // The flag to indicate if Intel RDT/CAT is enabled 185 catEnabled bool 186 // The flag to indicate if Intel RDT/MBA is enabled 187 mbaEnabled bool 188 189 // For Intel RDT initialization 190 initOnce sync.Once 191 192 errNotFound = errors.New("Intel RDT not available") 193 ) 194 195 // Check if Intel RDT sub-features are enabled in featuresInit() 196 func featuresInit() { 197 initOnce.Do(func() { 198 // 1. Check if Intel RDT "resource control" filesystem is available. 199 // The user guarantees to mount the filesystem. 200 root, err := Root() 201 if err != nil { 202 return 203 } 204 205 // 2. Check if Intel RDT sub-features are available in "resource 206 // control" filesystem. Intel RDT sub-features can be 207 // selectively disabled or enabled by kernel command line 208 // (e.g., rdt=!l3cat,mba) in 4.14 and newer kernel 209 if _, err := os.Stat(filepath.Join(root, "info", "L3")); err == nil { 210 catEnabled = true 211 } 212 if _, err := os.Stat(filepath.Join(root, "info", "MB")); err == nil { 213 mbaEnabled = true 214 } 215 if _, err := os.Stat(filepath.Join(root, "info", "L3_MON")); err != nil { 216 return 217 } 218 enabledMonFeatures, err = getMonFeatures(root) 219 if err != nil { 220 return 221 } 222 if enabledMonFeatures.mbmTotalBytes || enabledMonFeatures.mbmLocalBytes { 223 mbmEnabled = true 224 } 225 if enabledMonFeatures.llcOccupancy { 226 cmtEnabled = true 227 } 228 }) 229 } 230 231 // findIntelRdtMountpointDir returns the mount point of the Intel RDT "resource control" filesystem. 232 func findIntelRdtMountpointDir() (string, error) { 233 mi, err := mountinfo.GetMounts(func(m *mountinfo.Info) (bool, bool) { 234 // similar to mountinfo.FSTypeFilter but stops after the first match 235 if m.FSType == "resctrl" { 236 return false, true // don't skip, stop 237 } 238 return true, false // skip, keep going 239 }) 240 if err != nil { 241 return "", err 242 } 243 if len(mi) < 1 { 244 return "", errNotFound 245 } 246 247 return mi[0].Mountpoint, nil 248 } 249 250 // For Root() use only. 251 var ( 252 intelRdtRoot string 253 intelRdtRootErr error 254 rootOnce sync.Once 255 ) 256 257 // The kernel creates this (empty) directory if resctrl is supported by the 258 // hardware and kernel. The user is responsible for mounting the resctrl 259 // filesystem, and they could mount it somewhere else if they wanted to. 260 const defaultResctrlMountpoint = "/sys/fs/resctrl" 261 262 // Root returns the Intel RDT "resource control" filesystem mount point. 263 func Root() (string, error) { 264 rootOnce.Do(func() { 265 // Does this system support resctrl? 266 var statfs unix.Statfs_t 267 if err := unix.Statfs(defaultResctrlMountpoint, &statfs); err != nil { 268 if errors.Is(err, unix.ENOENT) { 269 err = errNotFound 270 } 271 intelRdtRootErr = err 272 return 273 } 274 275 // Has the resctrl fs been mounted to the default mount point? 276 if statfs.Type == unix.RDTGROUP_SUPER_MAGIC { 277 intelRdtRoot = defaultResctrlMountpoint 278 return 279 } 280 281 // The resctrl fs could have been mounted somewhere nonstandard. 282 intelRdtRoot, intelRdtRootErr = findIntelRdtMountpointDir() 283 }) 284 285 return intelRdtRoot, intelRdtRootErr 286 } 287 288 // Gets a single uint64 value from the specified file. 289 func getIntelRdtParamUint(path, file string) (uint64, error) { 290 fileName := filepath.Join(path, file) 291 contents, err := os.ReadFile(fileName) 292 if err != nil { 293 return 0, err 294 } 295 296 res, err := fscommon.ParseUint(string(bytes.TrimSpace(contents)), 10, 64) 297 if err != nil { 298 return res, fmt.Errorf("unable to parse %q as a uint from file %q", string(contents), fileName) 299 } 300 return res, nil 301 } 302 303 // Gets a string value from the specified file 304 func getIntelRdtParamString(path, file string) (string, error) { 305 contents, err := os.ReadFile(filepath.Join(path, file)) 306 if err != nil { 307 return "", err 308 } 309 310 return string(bytes.TrimSpace(contents)), nil 311 } 312 313 func writeFile(dir, file, data string) error { 314 if dir == "" { 315 return fmt.Errorf("no such directory for %s", file) 316 } 317 if err := os.WriteFile(filepath.Join(dir, file), []byte(data+"\n"), 0o600); err != nil { 318 return newLastCmdError(fmt.Errorf("intelrdt: unable to write %v: %w", data, err)) 319 } 320 return nil 321 } 322 323 // Get the read-only L3 cache information 324 func getL3CacheInfo() (*L3CacheInfo, error) { 325 l3CacheInfo := &L3CacheInfo{} 326 327 rootPath, err := Root() 328 if err != nil { 329 return l3CacheInfo, err 330 } 331 332 path := filepath.Join(rootPath, "info", "L3") 333 cbmMask, err := getIntelRdtParamString(path, "cbm_mask") 334 if err != nil { 335 return l3CacheInfo, err 336 } 337 minCbmBits, err := getIntelRdtParamUint(path, "min_cbm_bits") 338 if err != nil { 339 return l3CacheInfo, err 340 } 341 numClosids, err := getIntelRdtParamUint(path, "num_closids") 342 if err != nil { 343 return l3CacheInfo, err 344 } 345 346 l3CacheInfo.CbmMask = cbmMask 347 l3CacheInfo.MinCbmBits = minCbmBits 348 l3CacheInfo.NumClosids = numClosids 349 350 return l3CacheInfo, nil 351 } 352 353 // Get the read-only memory bandwidth information 354 func getMemBwInfo() (*MemBwInfo, error) { 355 memBwInfo := &MemBwInfo{} 356 357 rootPath, err := Root() 358 if err != nil { 359 return memBwInfo, err 360 } 361 362 path := filepath.Join(rootPath, "info", "MB") 363 bandwidthGran, err := getIntelRdtParamUint(path, "bandwidth_gran") 364 if err != nil { 365 return memBwInfo, err 366 } 367 delayLinear, err := getIntelRdtParamUint(path, "delay_linear") 368 if err != nil { 369 return memBwInfo, err 370 } 371 minBandwidth, err := getIntelRdtParamUint(path, "min_bandwidth") 372 if err != nil { 373 return memBwInfo, err 374 } 375 numClosids, err := getIntelRdtParamUint(path, "num_closids") 376 if err != nil { 377 return memBwInfo, err 378 } 379 380 memBwInfo.BandwidthGran = bandwidthGran 381 memBwInfo.DelayLinear = delayLinear 382 memBwInfo.MinBandwidth = minBandwidth 383 memBwInfo.NumClosids = numClosids 384 385 return memBwInfo, nil 386 } 387 388 // Get diagnostics for last filesystem operation error from file info/last_cmd_status 389 func getLastCmdStatus() (string, error) { 390 rootPath, err := Root() 391 if err != nil { 392 return "", err 393 } 394 395 path := filepath.Join(rootPath, "info") 396 lastCmdStatus, err := getIntelRdtParamString(path, "last_cmd_status") 397 if err != nil { 398 return "", err 399 } 400 401 return lastCmdStatus, nil 402 } 403 404 // WriteIntelRdtTasks writes the specified pid into the "tasks" file 405 func WriteIntelRdtTasks(dir string, pid int) error { 406 if dir == "" { 407 return fmt.Errorf("no such directory for %s", intelRdtTasks) 408 } 409 410 // Don't attach any pid if -1 is specified as a pid 411 if pid != -1 { 412 if err := os.WriteFile(filepath.Join(dir, intelRdtTasks), []byte(strconv.Itoa(pid)), 0o600); err != nil { 413 return newLastCmdError(fmt.Errorf("intelrdt: unable to add pid %d: %w", pid, err)) 414 } 415 } 416 return nil 417 } 418 419 // Check if Intel RDT/CAT is enabled 420 func IsCATEnabled() bool { 421 featuresInit() 422 return catEnabled 423 } 424 425 // Check if Intel RDT/MBA is enabled 426 func IsMBAEnabled() bool { 427 featuresInit() 428 return mbaEnabled 429 } 430 431 // Get the path of the clos group in "resource control" filesystem that the container belongs to 432 func (m *Manager) getIntelRdtPath() (string, error) { 433 rootPath, err := Root() 434 if err != nil { 435 return "", err 436 } 437 438 clos := m.id 439 if m.config.IntelRdt != nil && m.config.IntelRdt.ClosID != "" { 440 clos = m.config.IntelRdt.ClosID 441 } 442 443 return filepath.Join(rootPath, clos), nil 444 } 445 446 // Applies Intel RDT configuration to the process with the specified pid 447 func (m *Manager) Apply(pid int) (err error) { 448 // If intelRdt is not specified in config, we do nothing 449 if m.config.IntelRdt == nil { 450 return nil 451 } 452 453 path, err := m.getIntelRdtPath() 454 if err != nil { 455 return err 456 } 457 458 m.mu.Lock() 459 defer m.mu.Unlock() 460 461 if m.config.IntelRdt.ClosID != "" && m.config.IntelRdt.L3CacheSchema == "" && m.config.IntelRdt.MemBwSchema == "" { 462 // Check that the CLOS exists, i.e. it has been pre-configured to 463 // conform with the runtime spec 464 if _, err := os.Stat(path); err != nil { 465 return fmt.Errorf("clos dir not accessible (must be pre-created when l3CacheSchema and memBwSchema are empty): %w", err) 466 } 467 } 468 469 if err := os.MkdirAll(path, 0o755); err != nil { 470 return newLastCmdError(err) 471 } 472 473 if err := WriteIntelRdtTasks(path, pid); err != nil { 474 return newLastCmdError(err) 475 } 476 477 m.path = path 478 return nil 479 } 480 481 // Destroys the Intel RDT container-specific 'container_id' group 482 func (m *Manager) Destroy() error { 483 // Don't remove resctrl group if closid has been explicitly specified. The 484 // group is likely externally managed, i.e. by some other entity than us. 485 // There are probably other containers/tasks sharing the same group. 486 if m.config.IntelRdt != nil && m.config.IntelRdt.ClosID == "" { 487 m.mu.Lock() 488 defer m.mu.Unlock() 489 if err := os.RemoveAll(m.GetPath()); err != nil { 490 return err 491 } 492 m.path = "" 493 } 494 return nil 495 } 496 497 // Returns Intel RDT path to save in a state file and to be able to 498 // restore the object later 499 func (m *Manager) GetPath() string { 500 if m.path == "" { 501 m.path, _ = m.getIntelRdtPath() 502 } 503 return m.path 504 } 505 506 // Returns statistics for Intel RDT 507 func (m *Manager) GetStats() (*Stats, error) { 508 // If intelRdt is not specified in config 509 if m.config.IntelRdt == nil { 510 return nil, nil 511 } 512 513 m.mu.Lock() 514 defer m.mu.Unlock() 515 stats := newStats() 516 517 rootPath, err := Root() 518 if err != nil { 519 return nil, err 520 } 521 // The read-only L3 cache and memory bandwidth schemata in root 522 tmpRootStrings, err := getIntelRdtParamString(rootPath, "schemata") 523 if err != nil { 524 return nil, err 525 } 526 schemaRootStrings := strings.Split(tmpRootStrings, "\n") 527 528 // The L3 cache and memory bandwidth schemata in container's clos group 529 containerPath := m.GetPath() 530 tmpStrings, err := getIntelRdtParamString(containerPath, "schemata") 531 if err != nil { 532 return nil, err 533 } 534 schemaStrings := strings.Split(tmpStrings, "\n") 535 536 if IsCATEnabled() { 537 // The read-only L3 cache information 538 l3CacheInfo, err := getL3CacheInfo() 539 if err != nil { 540 return nil, err 541 } 542 stats.L3CacheInfo = l3CacheInfo 543 544 // The read-only L3 cache schema in root 545 for _, schemaRoot := range schemaRootStrings { 546 if strings.Contains(schemaRoot, "L3") { 547 stats.L3CacheSchemaRoot = strings.TrimSpace(schemaRoot) 548 } 549 } 550 551 // The L3 cache schema in container's clos group 552 for _, schema := range schemaStrings { 553 if strings.Contains(schema, "L3") { 554 stats.L3CacheSchema = strings.TrimSpace(schema) 555 } 556 } 557 } 558 559 if IsMBAEnabled() { 560 // The read-only memory bandwidth information 561 memBwInfo, err := getMemBwInfo() 562 if err != nil { 563 return nil, err 564 } 565 stats.MemBwInfo = memBwInfo 566 567 // The read-only memory bandwidth information 568 for _, schemaRoot := range schemaRootStrings { 569 if strings.Contains(schemaRoot, "MB") { 570 stats.MemBwSchemaRoot = strings.TrimSpace(schemaRoot) 571 } 572 } 573 574 // The memory bandwidth schema in container's clos group 575 for _, schema := range schemaStrings { 576 if strings.Contains(schema, "MB") { 577 stats.MemBwSchema = strings.TrimSpace(schema) 578 } 579 } 580 } 581 582 if IsMBMEnabled() || IsCMTEnabled() { 583 err = getMonitoringStats(containerPath, stats) 584 if err != nil { 585 return nil, err 586 } 587 } 588 589 return stats, nil 590 } 591 592 // Set Intel RDT "resource control" filesystem as configured. 593 func (m *Manager) Set(container *configs.Config) error { 594 // About L3 cache schema: 595 // It has allocation bitmasks/values for L3 cache on each socket, 596 // which contains L3 cache id and capacity bitmask (CBM). 597 // Format: "L3:<cache_id0>=<cbm0>;<cache_id1>=<cbm1>;..." 598 // For example, on a two-socket machine, the schema line could be: 599 // L3:0=ff;1=c0 600 // which means L3 cache id 0's CBM is 0xff, and L3 cache id 1's CBM 601 // is 0xc0. 602 // 603 // The valid L3 cache CBM is a *contiguous bits set* and number of 604 // bits that can be set is less than the max bit. The max bits in the 605 // CBM is varied among supported Intel CPU models. Kernel will check 606 // if it is valid when writing. e.g., default value 0xfffff in root 607 // indicates the max bits of CBM is 20 bits, which mapping to entire 608 // L3 cache capacity. Some valid CBM values to set in a group: 609 // 0xf, 0xf0, 0x3ff, 0x1f00 and etc. 610 // 611 // 612 // About memory bandwidth schema: 613 // It has allocation values for memory bandwidth on each socket, which 614 // contains L3 cache id and memory bandwidth. 615 // Format: "MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;..." 616 // For example, on a two-socket machine, the schema line could be: 617 // "MB:0=20;1=70" 618 // 619 // The minimum bandwidth percentage value for each CPU model is 620 // predefined and can be looked up through "info/MB/min_bandwidth". 621 // The bandwidth granularity that is allocated is also dependent on 622 // the CPU model and can be looked up at "info/MB/bandwidth_gran". 623 // The available bandwidth control steps are: min_bw + N * bw_gran. 624 // Intermediate values are rounded to the next control step available 625 // on the hardware. 626 // 627 // If MBA Software Controller is enabled through mount option 628 // "-o mba_MBps": mount -t resctrl resctrl -o mba_MBps /sys/fs/resctrl 629 // We could specify memory bandwidth in "MBps" (Mega Bytes per second) 630 // unit instead of "percentages". The kernel underneath would use a 631 // software feedback mechanism or a "Software Controller" which reads 632 // the actual bandwidth using MBM counters and adjust the memory 633 // bandwidth percentages to ensure: 634 // "actual memory bandwidth < user specified memory bandwidth". 635 // 636 // For example, on a two-socket machine, the schema line could be 637 // "MB:0=5000;1=7000" which means 5000 MBps memory bandwidth limit on 638 // socket 0 and 7000 MBps memory bandwidth limit on socket 1. 639 if container.IntelRdt != nil { 640 path := m.GetPath() 641 l3CacheSchema := container.IntelRdt.L3CacheSchema 642 memBwSchema := container.IntelRdt.MemBwSchema 643 644 // TODO: verify that l3CacheSchema and/or memBwSchema match the 645 // existing schemata if ClosID has been specified. This is a more 646 // involved than reading the file and doing plain string comparison as 647 // the value written in does not necessarily match what gets read out 648 // (leading zeros, cache id ordering etc). 649 650 // Write a single joint schema string to schemata file 651 if l3CacheSchema != "" && memBwSchema != "" { 652 if err := writeFile(path, "schemata", l3CacheSchema+"\n"+memBwSchema); err != nil { 653 return err 654 } 655 } 656 657 // Write only L3 cache schema string to schemata file 658 if l3CacheSchema != "" && memBwSchema == "" { 659 if err := writeFile(path, "schemata", l3CacheSchema); err != nil { 660 return err 661 } 662 } 663 664 // Write only memory bandwidth schema string to schemata file 665 if l3CacheSchema == "" && memBwSchema != "" { 666 if err := writeFile(path, "schemata", memBwSchema); err != nil { 667 return err 668 } 669 } 670 } 671 672 return nil 673 } 674 675 func newLastCmdError(err error) error { 676 status, err1 := getLastCmdStatus() 677 if err1 == nil { 678 return fmt.Errorf("%w, last_cmd_status: %s", err, status) 679 } 680 return err 681 } 682