Windows Subsystem for Linux (WSL) on Windows 11

WSL or Windows Subsystem for Linux is basically a software layer in Windows (as of this writing, I only have Windows 10 running on my work laptop). This layer serves to allow you to run Linux binaries natively on Windows.

And I just got myself a new laptop with Windows 11 Home Edition. Enabling WSL on Windows 10 will be done a bit differently.

This layer also provides an environment that includes the Linux kernel, command-line tools and software packages enabling you to run Linux utilities on Windows.

Think of it similarly to Oracle VM Virtualbox. This something I'm use to particularly which allows me to setup a sandbox environment.

To start, open PowerShell as an administrator (you can do this by right clicking the PowerShell and selecting "Run as Administrator"), and enter the command wsl --install.

PS C:\Users\shahmatd> wsl --install

And you will be prompted to restart your machine. After restart, set the default WSL to version 2.

PS C:\Users\shahmatd> wsl --set-default-version 2 

For this instance, I will setup the latest fedora version 37. Fedora 37 or any version isn't readily made available on WSL. How do I know this? The following command lists the linux distribution that are made readily available on WSL.

PS C:\Windows\system32> wsl --list -o

The following is a list of valid distributions that can be installed.

Install using 'wsl.exe --install <Distro>'.

NAME                                   FRIENDLY NAME

Ubuntu                                 Ubuntu

Debian                                 Debian GNU/Linux

kali-linux                             Kali Linux Rolling

Ubuntu-18.04                           Ubuntu 18.04 LTS

Ubuntu-20.04                           Ubuntu 20.04 LTS

Ubuntu-22.04                           Ubuntu 22.04 LTS

OracleLinux_8_5                        Oracle Linux 8.5

OracleLinux_7_9                        Oracle Linux 7.9

SUSE-Linux-Enterprise-Server-15-SP4    SUSE Linux Enterprise Server 15 SP4

openSUSE-Leap-15.4                     openSUSE Leap 15.4

openSUSE-Tumbleweed                    openSUSE Tumbleweed

Fire up a web browser and open the following link. Pick the x86_64 architecture, download the respective .tar.xz. Extract using 7zip. Extract the downloaded .tar.xz and you will get another tar file, which you will need to extract a file called layer.tar.

From the PowerShell prompt, create a directory under the home directory of the account you're currently logged on as:

PS C:\Users\shahmatd> mkdir $HOME\wsl\fedora

Import the layer.tar file using the wsl command line by running:

PS C:\Users\shahmatd> wsl --import fedora $HOME\wsl\fedora 'C:\Users\shahmatd\Downloads\New folder\65462375a5e75e333a8988a5c08ede73131b0ce6eef5f249c5893d1d41394ed7\layer.tar'

Import in progress, this may take a few minutes.

The operation completed successfully.

Once import is successful, list the installed distribution as follows:

PS C:\Users\shahmatd> wsl --list -v

NAME STATE VERSION

* fedora Stopped 2

To start the fedora distribution, run:

PS C:\Users\shahmatd> wsl -d fedora

[root@Marsupalami85 ~]#

Press Ctrl+D to exit to the PowerShell prompt

To confirm the fedora distribution has started properly

PS C:\Windows\system32> wsl --list -v

NAME STATE VERSION

* fedora Running 2

Post Installation Steps

Launch the recently installed fedora installation.

Try updating the fedora OS with the latest packages. You will notice that you will be logged as root as denoted by the shell prompt:

[root@Marsupalami85 ~]# dnf update

Fedora 37 - x86_64                                                                      3.4 MB/s |  82 MB     00:23

Fedora 37 openh264 (From Cisco) - x86_64                                                993  B/s | 2.5 kB     00:02

Fedora Modular 37 - x86_64                                                              1.6 MB/s | 3.8 MB     00:02

Fedora 37 - x86_64 - Updates                                                            3.1 MB/s |  28 MB     00:09

Fedora Modular 37 - x86_64 - Updates                                                    645 kB/s | 2.9 MB     00:04

Dependencies resolved.

Nothing to do.

Complete!

The next thing to do would be to create a non-privilege user, but before you can do that install the following packages:

[root@Marsupalami85 ~]# dnf install -y util-linux passwd cracklib-dicts

Last metadata expiration check: 0:03:46 ago on Mon Apr 17 23:10:17 2023.

Dependencies resolved.

========================================================================================================================

 Package                         Architecture            Version                          Repository               Size

========================================================================================================================

Installing:

 cracklib-dicts                  x86_64                  2.9.7-30.fc37                    fedora                  3.6 M

 passwd                          x86_64                  0.80-13.fc37                     fedora                  121 k

 util-linux                      x86_64                  2.38.1-1.fc37                    fedora                  2.3 M

Installing dependencies:

 libfdisk                        x86_64                  2.38.1-1.fc37                    fedora                  160 k

 libuser                         x86_64                  0.63-13.fc37                     fedora                  399 k

 libutempter                     x86_64                  1.2.1-7.fc37                     fedora                   26 k

Transaction Summary

========================================================================================================================

Install  6 Packages

Total download size: 6.7 M

Installed size: 23 M

Downloading Packages:

(1/6): libfdisk-2.38.1-1.fc37.x86_64.rpm                                                150 kB/s | 160 kB     00:01

(2/6): libutempter-1.2.1-7.fc37.x86_64.rpm                                              157 kB/s |  26 kB     00:00

(3/6): passwd-0.80-13.fc37.x86_64.rpm                                                   577 kB/s | 121 kB     00:00

(4/6): libuser-0.63-13.fc37.x86_64.rpm                                                  274 kB/s | 399 kB     00:01

(5/6): cracklib-dicts-2.9.7-30.fc37.x86_64.rpm                                          1.5 MB/s | 3.6 MB     00:02

(6/6): util-linux-2.38.1-1.fc37.x86_64.rpm                                              1.3 MB/s | 2.3 MB     00:01

------------------------------------------------------------------------------------------------------------------------

Total                                                                                   1.5 MB/s | 6.7 MB     00:04

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

  Preparing        :                                                                                                1/1

  Running scriptlet: libutempter-1.2.1-7.fc37.x86_64                                                                1/6

  Installing       : libutempter-1.2.1-7.fc37.x86_64                                                                1/6

  Installing       : libuser-0.63-13.fc37.x86_64                                                                    2/6

  Installing       : libfdisk-2.38.1-1.fc37.x86_64                                                                  3/6

  Installing       : util-linux-2.38.1-1.fc37.x86_64                                                                4/6

warning: /etc/adjtime created as /etc/adjtime.rpmnew

  Installing       : passwd-0.80-13.fc37.x86_64                                                                     5/6

  Installing       : cracklib-dicts-2.9.7-30.fc37.x86_64                                                            6/6

  Running scriptlet: cracklib-dicts-2.9.7-30.fc37.x86_64                                                            6/6

/usr/sbin/ldconfig: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link

  Verifying        : cracklib-dicts-2.9.7-30.fc37.x86_64                                                            1/6

  Verifying        : libfdisk-2.38.1-1.fc37.x86_64                                                                  2/6

  Verifying        : libuser-0.63-13.fc37.x86_64                                                                    3/6

  Verifying        : libutempter-1.2.1-7.fc37.x86_64                                                                4/6

  Verifying        : passwd-0.80-13.fc37.x86_64                                                                     5/6

  Verifying        : util-linux-2.38.1-1.fc37.x86_64                                                                6/6

Installed:

  cracklib-dicts-2.9.7-30.fc37.x86_64        libfdisk-2.38.1-1.fc37.x86_64        libuser-0.63-13.fc37.x86_64

  libutempter-1.2.1-7.fc37.x86_64            passwd-0.80-13.fc37.x86_64           util-linux-2.38.1-1.fc37.x86_64

Complete!

Once those packages has been installed create your first user account. In this case, let's call it shahmatd. Set an appropriate password for the newly created user account and assign a secondary group called 'wheel' to it.

[root@Marsupalami85 ~]# useradd shahmatd

[root@Marsupalami85 ~]# passwd shahmatd

Changing password for user shahmatd.

New password:

Retype new password:

passwd: all authentication tokens updated successfully.

[root@Marsupalami85 system32]# usermod -G wheel shahmatd

Log yourself out and back in but instead of logging on as root, log on as the non-privileged user you had just created.

PS C:\Windows\system32> wsl -d fedora -u shahmatd

[shahmatd@Marsupalami85 ~]$

To make logging on via WSL as the user shahmatd as the default behavior:

[shahmatd@Marsupalami85 ~]# printf "\n[user]\ndefault = username\n" | sudo tee -a /etc/wsl.conf

Once this is done, log out and log yourself back in but this time, you won't be supplying the username you'll be logging on as, only issue:

PS C:\Windows\system32> wsl -d fedora

[shahmatd@Marsupalami85 system32]$

Next, we will revisit this fedora distribution, and enabled systemd, and ssh server.

Cloning a RHEL/CentOS 7.x onto a bootable second disk

This is a little one of those things where you just have to do it yourself.

Basically the requirement would be to have two disks on your system. I am testing this on an Oracle VM Virtualbox environment, with a CentOS 7.7 (1908) as a VM guest.

The rest of the VM guest setup is pretty straightforward nothing out of the ordinary, with two disks of the same size (or virtual disks) attached. This requires further testing done on a bare metal x86 server, which I would hope to be able to do soon with EFI.

The first disk is /dev/sda, and the second disk is /dev/sdb.

The second disk is basically empty. And obviously the first one isn't. And this is how the partition table looks like:

# parted /dev/sda p
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1075MB  1074MB  primary  xfs          boot
 2      1075MB  34.4GB  33.3GB  primary               lvm

Partition 1 (or /dev/sda1) is a native partition, where as partition 2 (/dev/sda2) is under LVM. As you can see here:

# pvs
  PV         VG      Fmt  Attr PSize   PFree
  /dev/sda2  centos  lvm2 a--  <31 .01g="" 8.00m="" p="">
# vgs
  VG      #PV #LV #SN Attr   VSize   VFree
  centos    1   8   0 wz--n- <31 .01g="" 8.00m="" p="">
# lvs
  LV            VG     Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home          centos -wi-ao---- 5.00g
  root          centos -wi-ao---- 5.00g
  swap          centos -wi-ao---- 2.00g
  tmp           centos -wi-ao---- 3.00g
  var           centos -wi-ao---- 5.00g
  var_log       centos -wi-ao---- 5.00g
  var_log_audit centos -wi-ao---- 3.00g
  var_tmp       centos -wi-ao---- 3.00g

The requirement is to basically to clone the contents of first disk (/dev/sda) to the second disk (/dev/sdb) which should include partition table, MBR or gpt, regardless.

And all filesystems are xfs, which is the default filesystem for all RHEL or CentOS 7.x install.

The steps

The fastest way to achieve this is to use the good ol' dd. I am lazy so, here's the command:

# dd if=/dev/sda of=/dev/sdb

You can add bs=512, or bs=1024 to make the dd cloning go faster (This is another topic altogether)

After cloning is complete, use parted to see if the exact same partition table is on the second disk (/dev/sdb). By right, you should be seeing:

# parted /dev/sdb p
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sdb: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1075MB  1074MB  primary  xfs          boot
 2      1075MB  34.4GB  33.3GB  primary               lvm

First things, first, test to see if /boot or /dev/sda1 is mountable. Before you do this you'll need to use xfs_repair (to zero the logs) in order to make it mountable:

# xfs_repair -L /dev/sda1

After that, the xfs file system on /dev/sda1 uuid needs to be regenerated because of the clone, where all uuid for the filesystems, PV and VG will be the same (duplicated) and only one can be activated at one time.

To do this:

# xfs_admin -U generate /dev/sda1

Once this is done, perform a test mount:

# mkdir -p /a/boot
# mount  /dev/sda1 /a/boot

Examine it's contents, by running :

# ls -l /a/boot
total 124568
-rw-r--r--. 1 root root   152980 Sep 30 22:23 config-3.10.0-1062.1.2.el7.x86_64
-rw-r--r--. 1 root root   151923 Jun 19 00:39 config-3.10.0-957.21.3.el7.x86_64
drwxr-xr-x. 3 root root       17 Jun 27 23:18 efi
drwxr-xr-x. 2 root root       27 Jun 27 23:19 grub
drwx------. 5 root root      132 Oct 29 23:56 grub2
-rw-------. 1 root root 57160365 Jun 27 23:24 initramfs-0-rescue-7ff3c24ba1924fe1af8443c0ea440407.img
-rw-------. 1 root root 21143881 Oct  3 18:41 initramfs-3.10.0-1062.1.2.el7.x86_64.img
-rw-------. 1 root root 21125579 Oct  3 18:42 initramfs-3.10.0-957.21.3.el7.x86_64.img
-rw-r--r--. 1 root root   318717 Sep 30 22:23 symvers-3.10.0-1062.1.2.el7.x86_64.gz
-rw-r--r--. 1 root root   314128 Jun 19 00:39 symvers-3.10.0-957.21.3.el7.x86_64.gz
-rw-------. 1 root root  3595191 Sep 30 22:23 System.map-3.10.0-1062.1.2.el7.x86_64
-rw-------. 1 root root  3545891 Jun 19 00:39 System.map-3.10.0-957.21.3.el7.x86_64
-rwxr-xr-x. 1 root root  6639904 Jun 27 23:24 vmlinuz-0-rescue-7ff3c24ba1924fe1af8443c0ea440407
-rwxr-xr-x. 1 root root  6734016 Sep 30 22:23 vmlinuz-3.10.0-1062.1.2.el7.x86_64
-rwxr-xr-x. 1 root root  6643904 Jun 19 00:39 vmlinuz-3.10.0-957.21.3.el7.x86_64

Unmount /a/boot.

# umount /a/boot

Perform the exact same thing with xfs_repair and xfs_admin on the LVs sitting on PV /dev/sdb2, which was cloned from /dev/sda2.

But before you can do that, you need to perform a VG import, but in this case, you will not be able to do so, because the uuid on the VG would have been duplicated (because of the dd clone).

Instead of using vgimport, use vgimportclone instead:

# vgimportclone --import -n centos1 /dev/sdb2

Where -n centos1 specifies the name of the new VG, because the original VG on /dev/sda2 (the source of the clone) is centos. /dev/sdb2 is it's PV.

Once you've done the above, you should be able to see both PVs and both VGs:

# pvs
  PV         VG      Fmt  Attr PSize   PFree
  /dev/sda2  centos  lvm2 a--  <31 .01g="" 8.00m="" p="">  /dev/sdb2  centos1 lvm2 a--  <31 .01g="" 8.00m="" p="">
# vgs
  VG      #PV #LV #SN Attr   VSize   VFree
  centos    1   8   0 wz--n- <31 .01g="" 8.00m="" p="">  centos1   1   8   0 wz--n- <31 .01g="" 8.00m="" p="">
Activate the VG centos1 if you are unable to access the LVs under it.

# vgchange -ay centos1
  8 logical volume(s) in volume group "centos1" now active

Once activated, you should be able to see all the LVs on both centos and centos1 VGs.

# lvs

  LV            VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert

  home          centos  -wi-ao---- 5.00g

  root          centos  -wi-ao---- 5.00g

  swap          centos  -wi-ao---- 2.00g

  tmp           centos  -wi-ao---- 3.00g

  var           centos  -wi-ao---- 5.00g

  var_log       centos  -wi-ao---- 5.00g

  var_log_audit centos  -wi-ao---- 3.00g

  var_tmp       centos  -wi-ao---- 3.00g

  home          centos1 -wi-a----- 5.00g

  root          centos1 -wi-a----- 5.00g

  swap          centos1 -wi-a----- 2.00g

  tmp           centos1 -wi-a----- 3.00g

  var           centos1 -wi-a----- 5.00g

  var_log       centos1 -wi-a----- 5.00g

  var_log_audit centos1 -wi-a----- 3.00g

  var_tmp       centos1 -wi-a----- 3.00g

The next thing for us to do is to ensure the LVs under the VG centos1 is mountable:

# for lvname in home root tmp var var_log var_log_audit var_tmp; do

> xfs_repair -L /dev/mapper/centos1-${lvname}
> xfs_admin -U generate /dev/mapper/centos1-${lvname}
> done

Once this is done, you should be able to mount these LVs. Remember the /a directory we created earlier. Mount all the relevant filesystems onto /a

# mount /dev/mapper/centos1-root /a
# mount /dev/sdb1 /a/boot
# mount /dev/mapper/centos1-home /a/home
# mount /dev/mapper/centos1-tmp /a/tmp
# mount /dev/mapper/centos1-var /a/var
# mount /dev/mapper/centos1-var_tmp /a/var/tmp
# mount /dev/mapper/centos1-var_log /a/var/log
# mount /dev/mapper/centos1-var_log_audit /a/var/log/audit
# mount -t proc none /a/proc
# mount -o bind /sys /a/sys
# mount -o bind /dev /a/dev

Chroot the /a by running

# chroot /a

Next find out what is the existing uuid for /dev/sda1 and /dev/sdb1.

# blkid | grep sd[ab]1
/dev/sda1: UUID="0f7eaa06-5707-45cc-99db-f3e507c44bcf" TYPE="xfs"
/dev/sdb1: UUID="9b6d885a-ea6d-4c28-997c-fb6f04dcec5e" TYPE="xfs"

Edit the contents of /a/etc/fstab, replace the UUID="<16 digit="" hexadecimal="">" from /dev/sda1 to /dev/sdb1 from the output above.

Change this:
UUID=0f7eaa06-5707-45cc-99db-f3e507c44bcf /boot                   xfs     defaults        0 0

To this:
UUID=9b6d885a-ea6d-4c28-997c-fb6f04dcec5e /boot                   xfs     defaults        0 0

Edit the contents of /a/boot/grub2/grub.cfg and locate the line containing:

### BEGIN /etc/grub.d/10_linux ###

menuentry 'CentOS Linux (3.10.0-1062.1.2.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.1.2.el7.x86_64-advanced-18586e25-cb00-4ef5-b913-e0613ede485a' {

        load_video

        set gfxpayload=keep

        insmod gzio

        insmod part_msdos

        insmod xfs

        set root='hd0,msdos1'

        if [ x$feature_platform_search_hint = xy ]; then

          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  0f7eaa06-5707-45cc-99db-f3e507c44bcf

        else

          search --no-floppy --fs-uuid --set=root 0f7eaa06-5707-45cc-99db-f3e507c44bcf

        fi

        linux16 /vmlinuz-3.10.0-1062.1.2.el7.x86_64 root=/dev/mapper/centos1-root ro crashkernel=auto rd.lvm.lv=centos1/root rd.lvm.lv=centos1/swap rhgb quiet

        initrd16 /initramfs-3.10.0-1062.1.2.el7.x86_64.img

}

Replace the uuid marked in yellow to 9b6d885a-ea6d-4c28-997c-fb6f04dcec5e

Amend the vg name from centos to centos1 in the file /a/etc/default/grub. Ensure these lines containing:

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet"

is changed to :

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos1/root rd.lvm.lv=centos/swap rhgb quiet"

Update the grub.cfg file under /a/etc/grub2/grub.cfg 

# grub2-mkconfig -o /boot/grub2/grub.cfg

Perform a touch relabel to relabel all selinux context.

# touch /.relable

Exit from the chroot /a environment:

# exit

And now you should be back to the environment on the first disk.

Remember the /a/boot/grub2/grub.cfg entry you modified under the chroot /a environment ?

Use the same entries and edit the first disk's /boot/grub2/grub.cfg

And add the following entry, this entry is similar to what we have on the second disk (which we chroot /a into earlier)

menuentry 'CentOS Linux (3.10.0-1062.1.2.el7.x86_64) 7 (Core) on Disk 2' --class centos --cla                                                                           ss gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062                                                                           .1.2.el7.x86_64-advanced-18586e25-cb00-4ef5-b913-e0613ede485a' {

        load_video

        set gfxpayload=keep

        insmod gzio

        insmod part_msdos

        insmod xfs

        set root='hd1,msdos1'

        if [ x$feature_platform_search_hint = xy ]; then

          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos                                                                           1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  9b6d885a-ea6d-4c28-997c-fb6f04dcec5e

        else

          search --no-floppy --fs-uuid --set=root 9b6d885a-ea6d-4c28-997c-fb6f04dcec5e

        fi

        linux16 /vmlinuz-3.10.0-1062.1.2.el7.x86_64 root=/dev/mapper/centos1-root ro crashker                                                                           nel=auto rd.lvm.lv=centos1/root rd.lvm.lv=centos1/swap rhgb quiet

        initrd16 /initramfs-3.10.0-1062.1.2.el7.x86_64.img

}

Save the file /boot/grub2/grub.cfg. And finally reboot to test.

Once reboot, your grub menu should look like this:

And all the above steps can be scripted. No problem.

MySQL Cluster

I was recently given the chance to try out MySQL Cluster by Oracle/MySQL. I've never tried configuring one before, and I thought was never going to find any need to. I suppose I may have told myself, who would've thought I'd be doing one now.

First things first, register for an Oracle Account on the Oracle Network. Simply visit http://edelivery.oracle.com. This should redirect you to a page where it will prompt to see if you'd like to login (if you happened to have an Oracle Account) or create a new one.

I already have a personal Oracle Account account which I had used my gmail e-mail address as it's username.

Once you've logged yourself in, the Oracle's E-Delivery portal will have a search text field, enter "Mysql cluster" and this search field will automatically return you all the available text match it could find and simply pick MySQL Cluster 7.4.1.1. To the right of the search text field is the pull down list that allows to select which platform or rather which binary format you'd like to download. I've picked x86_64 for my test.

Below the search text field, you should be able to see MySQL Cluster 7.4.1.1 as the selected item, click on the Continue button.

Expand the MySQL Cluster 7.4.1.1 item by clicking on the tiny triangle to the left of the select item. This will again prompt you to check the check box, I don't know it's asking you to confirm for the second time, I think it's unnecessary but hey, it's Oracle. You will see another Continue button, and I think you can already see the obvious pattern, especially on what to do next.

A popup box will appear with the list of available package format such as Debian, SuSE, Oracle/Linux and the ever handy tar ball format.

I'm lazy and since I'm on CentOS 6.8, I've picked RPM for Oracle Linux / RHEL 6 x86 (64-bit). Download accordingly, follow the next instructions.

Once download is complete, and if you're on x86_64 on CentOS 6.8 like I do, you will probably end up with a zip file called V138059-01.zip. Unzip this zip file and you will get a set of RPMs :

MySQL-Cluster-client-advanced-7.4.11-1.el6.x86_64.rpm
MySQL-Cluster-devel-advanced-7.4.11-1.el6.x86_64.rpm
MySQL-Cluster-embedded-advanced-7.4.11-1.el6.x86_64.rpm
MySQL-Cluster-server-advanced-7.4.11-1.el6.x86_64.rpm
MySQL-Cluster-shared-advanced-7.4.11-1.el6.x86_64.rpm
MySQL-Cluster-shared-compat-advanced-7.4.11-1.el6.x86_64.rpm
MySQL-Cluster-test-advanced-7.4.11-1.el6.x86_64.rpm

First things, first I have a rough diagram of how my MySQL cluster design looks like.

[Image here]

mgmt1 - Management Node 1

sql1 - SQL/API Node 1
sql2 - SQL/API Node 2
data1 - Data Node 1
data2 - Data Node 2

Management Node

First things first, on the designated management host, uninstall mysql-libs. mysql-libs would have some conflicting files when you'll attempt to install MySQL-Cluster-server-advanced.

# rpm -e --nodeps mysql-libs

Install the MySQL-Cluster-server-advanced, MySQL-Cluster-shared-advanced, and MySQL-Cluster-shared-compat-advanced RPM packages.

# rpm -Uvh MySQL-Cluster-server-advanced-7.4.11-1.el6.x86_64.rpm MySQL-Cluster-shared-compat-advanced-7.4.11-1.el6.x86_64.rpm MySQL-Cluster-shared-advanced-7.4.11-1.el6.x86_64.rpm


I have selected minimal install for every Linux install I've done so I can pick the packages that I would like to install. Always update to the latest kernel and all it's subsequent package updates.

wget - so I can grab packages from whichever site I'd like to
zip/unzip - to be able to unzip the MySQL Cluster zip file containing the RPM packages
numactl - can't remember why I needed this, aparrently it is a prerequisite
man - for obvious reasons, man pages
openssh-clients - didn't I say I picked minimal install earlier?
ntp - the ntp client, for system time synching
ntpdate - an ntp utility, out of habit I feel like I need for some reason
python-paramiko - a prerequisite if you intend to use MySQL Cluster Autoinstall (ndb_setup.py)


Create a directory called /var/lib/mysql-cluster/49

# mkdir -p /var/lib/mysql-cluster/49

Create a text file and call it config.ini under the directory /var/lib/mysql-cluster/49, with the following contents. The number 49 is the designated NDB node ID for the management host. This is all up to you.

================== config.ini begins =================
[NDB_MGMD DEFAULT]
Portnumber=1186

[NDB_MGMD]
NodeId=49
HostName=mgmt1
DataDir=/var/lib/mysql-cluster/49/
Portnumber=1186

[TCP DEFAULT]
SendBufferMemory=2M
ReceiveBufferMemory=2M

[NDBD DEFAULT]
BackupMaxWriteSize=1M
BackupDataBufferSize=16M
BackupLogBufferSize=4M
BackupMemory=20M
BackupReportFrequency=10
MemReportFrequency=30
LogLevelStartup=15
LogLevelShutdown=15
LogLevelCheckpoint=8
LogLevelNodeRestart=15
DataMemory=1M
IndexMemory=1M
MaxNoOfTables=4096
MaxNoOfTriggers=3500
NoOfReplicas=2
StringMemory=25
DiskPageBufferMemory=64M
SharedGlobalMemory=20M
LongMessageBuffer=32M
MaxNoOfConcurrentTransactions=16384
BatchSizePerLocalScan=512
FragmentLogFileSize=64M
NoOfFragmentLogFiles=16
RedoBuffer=32M
MaxNoOfExecutionThreads=2
StopOnError=false
LockPagesInMainMemory=1
TimeBetweenEpochsTimeout=32000
TimeBetweenWatchdogCheckInitial=60000
TransactionInactiveTimeout=60000
HeartbeatIntervalDbDb=15000
HeartbeatIntervalDbApi=15000

[NDBD]
NodeId=1
HostName=data1
DataDir=/var/lib/mysql-cluster/1/

[NDBD]
NodeId=2
HostName=data2
DataDir=/var/lib/mysql-cluster/2/

[MYSQLD DEFAULT]

[MYSQLD]
NodeId=55
HostName=sql1

[MYSQLD]
NodeId=56
HostName=sql2

[API]
NodeId=50
HostName=sql1

[API]
NodeId=52
HostName=sql2
================== config.ini ends =================

Data Nodes

On the designated data nodes, install the MySQL-Cluster-server-advanced, MySQL-Cluster-shared-advanced, and MySQL-Cluster-shared-compat-advanced RPM packages.

# rpm -Uvh MySQL-Cluster-server-advanced-7.4.11-1.el6.x86_64.rpm MySQL-Cluster-shared-compat-advanced-7.4.11-1.el6.x86_64.rpm MySQL-Cluster-shared-advanced-7.4.11-1.el6.x86_64.rpm

Create a directory called /var/lib/mysql-cluster/1

# mkdir -p /var/lib/mysql-cluster/1

SQL Nodes

While on the designated sql nodes, install the same MySQL-Cluster-server-advanced, MySQL-Cluster-shared-advanced, and MySQL-Cluster-shared-compat-advanced RPM packages.

# rpm -Uvh MySQL-Cluster-server-advanced-7.4.11-1.el6.x86_64.rpm MySQL-Cluster-shared-compat-advanced-7.4.11-1.el6.x86_64.rpm MySQL-Cluster-shared-advanced-7.4.11-1.el6.x86_64.rpm

Create a directory called /var/lib/mysql-cluster/55

# mkdir -p /var/lib/mysql-cluster/55

Initialize the necessary mysql databases and tables.

# mysql_install_db --basedir="/usr" --datadir="/var/lib/mysql-cluster/56"

Create a my.cnf under the /var/lib/mysql-cluster/56 directory

================== my.cnf begins =================
[mysqld]
log-error=mysqld.56.err
datadir="/var/lib/mysql-cluster/56/"
tmpdir="/var/lib/mysql-cluster/56/tmp"
basedir="/usr/"
port=3306
ndbcluster=on
ndb-nodeid=55
ndb-connectstring=mgmt1:1186,
socket="/var/lib/mysql-cluster/56/mysql.socket"
================== my.cnf ends =================

Edit the /etc/init.d/mysql, locate the directive other_args and amend accordingly as follows:

other_args="--ndbcluster --ndb-nodeid=56 --ndb-connectstring=mgmt1:1186"

Once all the configurations are done (in the following order) start the Management node, Data Nodes and followed by SQL Nodes.

Starting the Management Node

To start the Management Node run the following

# ndb_mgmd -f /var/lib/mysql-cluster/49/config.ini --initial

Starting the Data Nodes

To start the Data Nodes run the following instead:

# ndb_mgmd --ndb-node=1 --ndb-connectstring=mgmt1:1186 --initial

Starting the SQL Nodes

Lastly, start the SQL node by starting the MySQL server.

# /etc/init.d/mysql start

The Management Node client

From the Management Node, run the Management Node client, run show

# ndb_mgm

-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]    2 node(s)
id=1    @192.168.56.121  (mysql-5.6.29 ndb-7.4.11, Nodegroup: 0, *)
id=2    @192.168.56.122  (mysql-5.6.29 ndb-7.4.11, Nodegroup: 0)

[ndb_mgmd(MGM)]    1 node(s)
id=49    @192.168.56.101  (mysql-5.6.29 ndb-7.4.11)

[mysqld(API)]    4 node(s)
id=50 (not connected, accepting connect from sql1)
id=52 (not connected, accepting connect from sql2)
id=55    @192.168.56.111  (mysql-5.6.29 ndb-7.4.11)
id=56    @192.168.56.112  (mysql-5.6.29 ndb-7.4.11)

As you may notice my API nodes are not connected, and I am still looking for answers on this for the time being.

I guess I'll need to do some more reading on MySQL Cluster as it seems. The documentations are rather vague and even the instructions are not very clear to say the least.

Once all the services are up, and you'd have every intention to start creating your databases and tables using the NDB Cluster Storage engine. Simply change the table engine type to ndbcluster and you'll be pretty much on your way.

Shell specifically bash, how to generate random number between 1 to 4

I've had an assignment of which I thought I should be doing, which was to randomly pick four servers at a time, to pick either one of them and appropriately run script on the selected server. Server hostname format ends with a number between 1 to 4.

i=$(echo $(( ($RANDOM % 4 ) + 1)))
hostname=y3xa10${i}
echo $hostname

To try it out on a bash shell, what I got was basically:

# i=$(echo $(( ($RANDOM % 4 ) + 1))); echo y3xa10${i}
y3xa104
# i=$(echo $(( ($RANDOM % 4 ) + 1))); echo y3xa10${i}
y3xa102
# i=$(echo $(( ($RANDOM % 4 ) + 1))); echo y3xa10${i}
y3xa104
# i=$(echo $(( ($RANDOM % 4 ) + 1))); echo y3xa10${i}
y3xa102
# i=$(echo $(( ($RANDOM % 4 ) + 1))); echo y3xa10${i}
y3xa101
# i=$(echo $(( ($RANDOM % 4 ) + 1))); echo y3xa10${i}
y3xa104
# i=$(echo $(( ($RANDOM % 4 ) + 1))); echo y3xa10${i}
y3xa103
# i=$(echo $(( ($RANDOM % 4 ) + 1))); echo y3xa10${i}
y3xa102

I think this suits my requirement pretty much.

Linux - generating gpg key for rpm signing

Here how it goes. I wanted to create my own rpm for rootsh-1.5.3 after wanting to disable logging to syslog.

First things first, here's how you generate a gpg key.

[user@host~]$ gpg --gen-key
gpg (GnuPG) 1.4.5; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.

gpg: keyring `/home/xxxxxx/.gnupg/secring.gpg' created
gpg: keyring `/home/xxxxxx/.gnupg/pubring.gpg' created
Please select what kind of key you want:
   (1) DSA and Elgamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection? X
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is XXXX bits
Please specify how long the key should be valid.
         0 = key does not expire
       = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) XX
Key expires at Day DD Mmm YYYY HH:MM:SS AM MYT
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) "

Real name: Xxxxxx Xxxxxx
Email address: xxxxxx@xxxxx.xxx
Comment:
You selected this USER-ID:
    "Xxxxxx Xxxxxx "

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.


gpg: /home/xxxxxx/.gnupg/trustdb.gpg: trustdb created
gpg: key 2F74F867 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at YYYY-MM-DD
pub   2048R/2F74F867 2015-01-10 [expires: 2016-01-10]
      Key fingerprint = XXXX XXXX XXXX XXXX XXXX  XXXX XXXX XXXX XXXX XXXX 
uid                  Xxxxxx Xxxxxx

Note that this key cannot be used for encryption.  You may want to use
the command "--edit-key" to generate a subkey for this purpose.

And pick a long passphrase and remember them. The next time I will be doing would be to add a jpeg photo. It's actually unnecessary but I'll just doing for completeness sake. It's usually done when uploading gpg keys onto public key servers.

Downloaded a photo of myself and had it called me.jpg on my home directory.

[user@host ~]$ gpg --edit-key xxxxxx@xxxxx.xxx
gpg (GnuPG) 1.4.5; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.

Secret key is available.

pub  2048R/2F74F867  created: 2015-01-10  expires: 2016-01-10  usage: SC
                     trust: ultimate      validity: ultimate
[ultimate] (1). Xxxxxx Xxxxxx

Command> addphoto

Pick an image to use for your photo ID.  The image must be a JPEG file.
Remember that the image is stored within your public key.  If you use a
very large picture, your key will become very large as well!
Keeping the image close to 240x288 is a good size to use.

Enter JPEG filename for photo ID: /home/xxxxxx/me.jpg
This JPEG is really large (9181 bytes) !
Are you sure you want to use it? (y/N) n

Enter JPEG filename for photo ID: /home/xxxxxx/me.jpg
cannot open display:
Run 'eog --help' to see a full list of available command line options.
gpg: unable to display photo ID!
Is this photo correct (y/N/q)? y

You need a passphrase to unlock the secret key for
user: "Xxxxxx Xxxxxx "
XXXX-bit RSA key, ID XXXXXX, created YYYY-MM-DD


pub  XXXXX/XXXXXXXX  created: YYYY-MM-DD  expires: YYYY-MM-DD  usage: SC
                     trust: ultimate      validity: ultimate
[ultimate] (1). Xxxxxx Xxxxxx
[ unknown] (2)  [jpeg image of size 4936]

Command> save

To list of keys all available keys.

[user@host ~]$ gpg  --list-keys --fingerprint

/home/user/.gnupg/pubring.gpg

---------------------------------

pub   XXXXX/XXXXXXXX YYYY-MM-DD[expires: YYYY-MM-DD]

      Key fingerprint = XXXX XXXX XXXX XXXX XXXX  XXXX XXXX XXXX XXXX XXXX

uid                  Xxxxxx Xxxxxx 

uid                  [jpeg image of size 4936]

To list out the keys that has been installed:

[user@host ~]# rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n' 

Linux/Unix - ssh skip host key checkin and Known Hosts file

If you need to automate you remote scripting, like I do and without having being bothered to enter 'Yes' when being prompted to accept the key or with any other prompts you might see.

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ${user}@${host}

Linux - Extracting a single file from an RPM package

Create a temporary directory

# mkdir /tmp/1
# cd /tmp/1
# rpm2cpio /media/Packages/setup-2.8.14-20.el6_4.1.noarch.rpm | cpio -ivd

Copy the needed file out