When I first setup my Docker VM on my Proxmox machine I did it on Ubuntu Server 18.04 LTS with a desktop environment installed because I was still very new to using command line.  A desktop environment to manage everything with just seemed easier at the time.  I also just installed everything with 'docker run' commands and managed the images with Portainer.  After completing a hardware overhaul and moving my Proxmox instance from 1 2U server to 3 Intel NUCs in a cluster I decided it was time to finally migrate all my various containers(minus NGINX) to 1 Docker VM.  And since I'm much more comfortable working off a command line now compared to 2018 I could do this in the regular Ubuntu Server environment.

The Docker VM was upgraded to 20.04 TLS at some point without much issue so first attempt was to create a backup of the image in Proxmox and update it to 22.04 LTS...which did not go as planned.  Trying different methods either caused the install to fail or when it did complete I ran into an error many encountered in upgrading to 22.04 LTS that involves a conflict between network-manager and netplan.  22.04 seems to default to netplan even if your install was using network-manager before the upgrade and while I WAS eventually able to address it(using DHCP in netplan but static addresses were still not working) I opted to just start on a fresh 22.04 image in a different one and migrate everything over for a fresh start.

What are the benefits of docker-compose?

While not necessary it makes deploying multiple images a lot easier and helps in managing apps that depend on others to run properly.  Let's look at bookstack for example since it was initially one of the apps I was running.  If I wanted to run it as a docker image according to the documentation on its Docker Hub page it would look like this in the terminal

docker run -d \
  --name=bookstack \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Etc/UTC \
  -e APP_URL=<yourbaseurl> \
  -e DB_HOST=<yourdbhost> \
  -e DB_PORT=<yourdbport> \
  -e DB_USER=<yourdbuser> \
  -e DB_PASS=<yourdbpass> \
  -e DB_DATABASE=bookstackapp \
  -p 6875:80 \
  -v /path/to/data:/config \
  --restart unless-stopped \
  lscr.io/linuxserver/bookstack:latest

This will pull the image for the first run and sets up access to a database, but this is a command that will be hard to reference later unless you copy it somewhere else or have easy access to the terminal history of this server.  Also this assumes you have the database access already and doesn't automatically check to make sure the database is up before starting the image in the future to avoid errors.  Let's compare this to the suggested docker-compose yaml file excerpt from the same documentation

---
version: "2"
services:
  bookstack:
    image: lscr.io/linuxserver/bookstack
    container_name: bookstack
    environment:
      - PUID=1000
      - PGID=1000
      - APP_URL=https://bookstack.example.com
      - DB_HOST=bookstack_db
      - DB_PORT=3306
      - DB_USER=bookstack
      - DB_PASS=<yourdbpass>
      - DB_DATABASE=bookstackapp
    volumes:
      - ./bookstack_app_data:/config
    ports:
      - 6875:80
    restart: unless-stopped
    depends_on:
      - bookstack_db
  bookstack_db:
    image: lscr.io/linuxserver/mariadb
    container_name: bookstack_db
    environment:
      - PUID=1000
      - PGID=1000
      - MYSQL_ROOT_PASSWORD=<yourdbpass>
      - TZ=Europe/London
      - MYSQL_DATABASE=bookstackapp
      - MYSQL_USER=bookstack
      - MYSQL_PASSWORD=<yourdbpass>
    volumes:
      - ./bookstack_db_data:/config
    restart: unless-stopped

In this proposed docker-compose file that contains nothing else we see some similarities to the docker run command they provide but notice 2 things immediately.

1. This is easier to read in the yaml file compared to the docker run command.  Since the docker-compose.yaml file will exist on the filesystem as long as you don't remove it, being able to see where a mistake was possibly made when deploying it is much easier to parse in this format or reference in the future if you want to add additional services to this docker-compose file.
2. The depends_on key in the file allows you to reference another container.  If you don't have an external database and are going to be hosting 1 in a container specifically for bookstack you can include this command and it will make sure to wait for this container to start before starting the bookstack container.  

You can have multiple services in a docker-compose.yaml file too.  Mine currently has this Ghost blog(and a mysql database it depends on), Gitea and Bookstack(plus a mariadb database it depends on).  As long as there isn't any port conflicts it shouldn't cause any issues.  After creating the docker-compose.yaml file running the following command in the directory pulled images and started everything

docker-compose up

If I want to have the containers running in the background to free up the terminal can use this instead(preferred for me)

docker-compose up -d

Its the same as the --detach option.  If I want to bring all the images down I can run the following

docker-compose down

Do note that if you don't specify any volumes on the filesystem in the docker-compose file, doing this means you lose any data tied to that container.  The bookstack + bookstack database specified above have mapped volumes so this is avoided.  And updating images is as easy as either running

docker-compose pull

to update everything or if you wish to only update specific images(maybe you want to keep databases on specific versions) you can specify what image to pull by referencing the container name in the docker-compose.yaml file as seen below

docker-compose pull bookstack

This will only pull and update the bookstack service using the image reference in the compose file.  You can find references to these commands and other docker-compose commands at the official docker documentation here.

One final thing to note is that all these commands have to be run in the same directory the docker-compose.yaml file is located.  One way I approached this is to just have my default directory when I log into this server be the location where the docker-compose file is located.  You can achieve this by running the following command to append a line to your bashrc file

echo "cd /location/of/docker-compose" >> ~/.bashrc

This automatically puts me in the directory of the docker-compose.yaml file when I log in to save some time.  And incase I ever want to use a different docker-compose file for testing in another directory I could edit this to change it to that directory, this way my permanent containers don't get effected by any commands I might be running for managing temporary container deployments.

Using docker-compose has allowed me to neatly consolidate 2 servers running desktop environments down to 1 barebones Ubuntu server for most of my permanent containers, using less resources and making it much easier to manage via CLI which I'm more comfortable doing now than I was when I first started deploying these in 2018.

Wireguard is a VPN(virtual private network) protocol that allows you to connect to a network from a registered device outside of it.  Once the connection is established, it's as if you are browsing your LAN.  VPNs to home or business networks aren't anything new, but the recent rise in popularity and usage in VPNs, as well as Wireguards directly into the Linux Kernel starting with version 5.6 has brought even more attention to it over the last few months.  OpenVPN used to be the defacto VPN tunnel service used, but Wireguards performance blows it out of the water so far.  Here's a chart from vpnranks.com that goes over their analysis of the throughput difference between Wireguard and OpenVPN.

Picture taken from vpnranks.com, articled linked above
Wireguard can actually max out the gigabit connection without maxing out the CPU.  And anyone who has tried using OpenVPN knows that it's actually very CPU intensive, many people choose their pfsense builds based on CPU throughput under OpenVPN.  This is great for many reasons, as knowing that running a low power server on the cloud would not be bottlenecked by the CPU if you were using it as a VPN, meaning you can spin up a lower power server on the cloud for the same purpose and save alot of money over time.

So what's the best way to deploy Wireguard?  It usually depends on your setup.  If you are running a hypervisor like ESXI or Proxmox already, spinning up a VM dedicated to Wireguard is a viable option.  Another option is to deploy it on a separate device such as a Raspberry Pi or another type of small maker board.  Since as we've established Wireguard is not a CPU intensive process, the small ARM chips on the Pi are more than enough to handle this.

I had a Pi 3B+ that I grabbed on sale from Microcenter awhile back so I went with that.  It also supports USB booting out of the box which I like, so I formated a USB drive and after 2 restarts it booted right in.  After changing the password(never leave the default password on a Raspberry Pi) and changing the username to obscure it abit from whatever roaming bots that try to communicate with it since I will be opening a port for Wireguard, it was ready to go.

PiMyLifeUp has a very simple guide that I used that they recently published, and i'm gonna just link it here since it does a great job of just getting Wireguard setup on a Pi very quickly.  After setting it up and creating a profile to use on my Android phone, I grabbed the Android app and scanned the QR code to import the key.  After fixing a port forward error on my router where I opened a TCP port instead of UDP, my connection was made!

Wireguard is not perfect though.  Even it's creators have commented that it wasn't necessary designed with large commercial usage in mind.  VPN providers like NordVPN, ExpressVPN, PureVPN, etc.  There are concerns about privacy as by nature WireGuard does not dynamically assign IP addresses, so it logs and stores any IP address that connects to it.  What this means that if you were to use the NordVPN implementation they have to take extra measures to ensure your connecting IP is obscured, as your IP being permanently logged is a pretty bad idea when using a VPN who has servers you have no access or control over.  In this deployment we are using it less so to connect from our home network to the broad internet through a secure tunnel, but from the broad internet(in my case over a mobile network) to a home network through a secure tunnel.  As long as there are no vulnerabilities with Wireguard itself, it's as secure as my home network.

I ended up with an unexpected day off work yesterday, so I decided since I had some extra free time from coursework to finish up the rest of the days for the UpSkill challenge.

To finish up on the focus of logs that I've been making sure I get when completing this course, Day 18 covered log rotation.  By going into /etc/logrotate.conf you can see what the default values are for rotating system logs.  By default they are all global, but you also have the ability to specify specific rules for specific logs.  Certain logs I might not care to check more than once a week, but things like auth.log or nginx/apache access and error logs I would want daily snapshots of.  On this test server I didn't get that detailed, so the logrotate.conf for me looks like the image below.

I changed the first uncommented line from weekly to daily, uncommented dateext and added the dateformat line below it.  Note that you can format it however you wish, what I have up in dateformat is actually the default value if I left the rest of the link blank.  Set it however you prefer to read them.  You can see the last commented line on system-specific logs.  Let's see I only wanted to hold 1 weeks worth of auth.log files to reduce clutter.  I could add a specific configuration like this.

/var/log/auth.log {
    rotate 7
    mail example@example.com
}

First line shows that after counting 7 log files, it will simply delete the oldest one instead of rotating it down further.  This by itself will reduce clutter but it's also not exactly best practice, sometimes older logs can be important.  Instead of simply deleting it, by adding in the mail line(if we setup mail service on this server) any old logs that were to be purged are instead mailed to that email address.  This allows us to both reduce clutter in the directory but also maintain records of all these events on a seperate storage system.  Even with everything moving to the cloud, having some data stored locally or at a different service for redundancy is very important.

Cronjobs were the other major takeaway from the last few lessons.  Most good sysadmins are also lazy sysadmins, because they automate many tasks across many different servers and VMs.  Cronjobs are how they get to do that.  You write a script that then runs on a schedule.  I wouldn't need this for the log management example I just talked about, but what if I just want a list of recently disconnected ssh attempts done to my server run everyday for me?  I did a bash script last post about how to get that data and I put it in my path, a cronjob can be created that simply runs that for me once a day.

SFTP stands for SSH File Transfer Protocol, and it's a built in service for being able to send or receive files through SSH.  So if I want to grab files off my server without using automation, sftp is the way to go.  And if you setup your ssh connection in your config file already as I did, in my case connecting to my server is as easy as typing

sftp linuxskillup

and you are in the shell, ready to copy whatever you want.  The full manual page for sftp can be seen here, but basic commands will be using get to receive from the remote host, or put to sent to the remote host.  So if I want to copy over attacker.txt from this remote server to analyze it locally, I can simply type in the sftp shell

get /home/cleverness/attackers.txt /home/cleverness/Downloads

first path is the remote servers path, second is my home devices path.  If you are in the current directory of the file on the remote server, the full path isn't necessary, attackers.txt would have also worked since I was in the home directory there already.  And in reverse, if I want to move the file back after I examined it and dated it, in the sftp shell I'd go

put /home/cleverness/Downloads/attackers-09222020.txt /home/cleverness

Home path followed by remote path.  Nice way of securely transferring files between remote locations.

Overall I had a lot of fun with this challenge.  Some days were more informative than others for me since I know some stuff from running linux servers in my home already but I still learned alot.  Big thanks to snori74 on Github/Reddit for hosting the challenge.  I have the pages I did for this last post on the bottom, but here's the full Github page again for anyone interested in starting fresh.

Day 10 Day 11 Day 12 Day 13 Day 14 Day 15 Day 16 Day 17 Day 18 Day 19 Day 20

Day 7 involved installing Apache2 on my Ubuntu server, which in troubleshooting inadvertently ended up with me doing things in Day 9 until I figured out my error.  Installing Apache allows you to display a webpage and route traffic on your server through Ports 80 and/or 443 depending on whether you have SSL setup.  On Ubuntu 20.04 it's supposed to be a pretty simple install to get the sample page showing on your IP/Domain but I made an oversight as this was my host time hosting a cloud server.

My first step was to make sure ports were open to the internet, and really should have been my first clue as to the real solution.  Checking the firewall with

sudo ufw status

showed that the firewall was actually inactive by default on this installation.  Normally it should be showing as active and what ports are in the list and whether they are allowed, as seen here

cleverness@Linux-Skillup-Challenge-Ubuntu:~$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
Apache                     ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
Apache (v6)                ALLOW       Anywhere (v6) 

So in my infinite wisdom, I thought "maybe if the firewall is on, connections will work through port 80" even though I was connected remotely through SSH so that shouldn't be the case at all, and not how firewalls work, but I'm not smart sometimes.  So I went about finding out the best way to add everything to the firewall and turning it on without accidentally locking myself out, which is apparently an easy thing to do.  Instead of adding in the ports manually, which is completely viable, I ended up using the 'app list' command

cleverness@Linux-Skillup-Challenge-Ubuntu:~$ sudo ufw app list
Available applications:
  Apache
  Apache Full
  Apache Secure
  OpenSSH

Here we see available applications that can be added to the firewall based on what is installed on this system.  As it's very barebones there are literally only 2 listed.  OpenSSH is just 1 port and pretty obvious since I am remote, so I add that first

cleverness@Linux-Skillup-Challenge-Ubuntu:~$ sudo ufw allow OpenSSH

Then when it comes to Apache, it depends what I am going to be doing with it.  If I want to use it as a reverse proxy I will probably want to do with Apache Full to allow connections from both port 80 and port 443, or Apache Secure if I want to only allow traffic through 443.  As this is a temporary instance and most I'll be doing before it disappears is trying out some web templates I just selected Apache.

cleverness@Linux-Skillup-Challenge-Ubuntu:~$ sudo ufw allow Apache

Then I enabled the Firewall and hope I didnt mess anything up, because if I did I was about to get disconnected from my SSH session.

cleverness@Linux-Skillup-Challenge-Ubuntu:~$ sudo ufw enable

I wasn't disconnected so it worked, and the firewall status shows active as I displayed above earlier.  But the Apache default page still wasn't loading.  After some Google-fu and asking on Discord, it dawned on me that I never allowed traffic through port 80 on Azure itself.  Similar to port forwarding on your home router, if you don't open the ports for the VM on Azure they won't allow traffic through it, for security reasons.  Port 22 was created by default when I first spun the VM up as I requested it but it slipped my mind right after doing so that I might need to do this again.

Literally right after I added the inbound rule, the default page worked!  So that solved that dilemma for me.

Day 8 dealt with using different commands for log reading and management.  One of my 2 big takeaways was using tail -f /path/to/log allows for real-time log reading.  So for example if on my NGINX VM I want to see real-time logs for this website, I would go

tail -f /var/log/nginx/cleverness.tech.access.log

the terminal window will then continue running, showing me when the pages are being accessed.  As this site doesn't get a ton of traffic it's very easy to follow.  But what if you are using a remote or local environment and many attempts per minute are being made to access your server?  It's still possible to do this but being able to break it down and output it somewhere is very helpful as a System Administrator.  That's where grep, the second big takeaway from Day 8 came in.

grep allows you to search and filter words and phrases from files, but it can also be used to further narrow down a previous grep search.  So if after examining logs I see many failed attempts at logging in with different usernames, I can narrow it down with something like

grep "authenticating" /var/log/auth.log| grep -v "root"

It'll first filter the lines in auth.log with authenticating in them, and then further filter them down to ones that include root.  The Day 8 course that goes into how to format the output using the cut command, and outputting that output to a text file for future reading and analysis.

grep "authenticating" /var/log/auth.log| grep -v "root"| cut -f 10- -d" " > attackers.txt

Depending on the log you are viewing and the line layout you may have to adjust the values abit, but it will leave you with an output of only IP addresses which you can use for future analysis.  And since auth.log logs many different connection types with different message, you can just change the grep inputs and once you get the output you desire, append it to the end attackers.txt with >> instead of > for a large file.

Instead of cut there is another program called awk which is more difficult to use but gives you more control over where you are pulling out from a file if you know where it is.  I found a nice example off this Github page

grep -rhi 'invalid' /var/log/auth.log* | awk '{print $10}' | uniq | sort > ~/ips.txt

This will grab the IPs from the 10th field after only grabbing the lines containing invalid, only list uniq values to avoid repeats, and sort them and pipe the output out to ips.txt  I tried looking into using an alias for grep and alias are not really super friendly when it comes to accepting multiple inputs like I would think about using for this situation, so I ended up learning a bit abunch bash functions.

Bash functions are custom functions you can write that can be executed any time, and in my case I can also pass through some values into them if I write them that way.  First I need to open the file I need to edit for all this to work like so

nano ~/.bashrc

At the very bottom of the bash file is where I am going to add in my function.

authscan() {
        grep -rhi "$1" /var/log/auth.log* | awk '{print $10}' | uniq | sort 
}

If you look at the line for authscan() you can see it's very similar to the last grep line with a small difference; The "$1" is where I want to pass through a specific value when running the function.  As I am only going to pass in 1 grep argument I am going to only pass in 1, but if I wanted to scan for different logs I could add a "$2" where the log file path is defined.  However if doing so, you might need to append awk manually when calling the function as it's very precise where it prints from.  Like my Apache and NGINX logs start with the IP address right at the beginning of the line in the logs, while auth.log will have it somewhere else depending on the type of connection occurring.

After saving and exiting, run the following to reload bash without disconnecting you from your environment

exec bash

Now to test it, I simply run the following to make sure I get the same output as the full grep command from earlier.

authscan 'invalid' > ips2.txt

And I end up with the same text file.  So if I don't feel like hitting the up arrow 10-40 times to look for the grep command I can use bash functions on my actual servers to get the same result.  The grep code I found isn't 100% perfect in filtering out everything that isn't an IP due to how wildly different the sentence structure is in auth.log, but in the apache2 access log it actually works flawlessly since the first field on each line in those logs is the connecting IP.  So for my NGINX VM by swiping the 10 for 1 in the print portion I get a much cleaner list of IPs in my output.

Github Day 7 Day 8 Day 9

Days 4 through 6 covers how to actually install applications remotely, some quality of life commands to make using the command line better, and vim.  Since these are things I have some familiarity with I figured it'd be best to go over vimtutor here as opposed to in a 100DaysofCode post when I start that up again as it is relevant.

Day 4 goes through how to use the apt command in Ubuntu to install applications from the official repositories.  Whichever linux distribution you install will have a list of repositories installed by default.  You can browse the official Ubuntu repository here, either by selecting the release you are on and manually searching if you know what you want already.  Very useful if you need something specific like a python debugger and want to see what applications Canonical allows in their repositories.  There's also the option of adding in extra repositories for access to other programs.  When using Fedora this was actually encouraged, as the RPM Fusion community repository contains a large amount of third-party applications that aren't in the main Fedora one.

Day 5 goes through some quality of life techniques to using the command line, the biggest one for me is tab to autocomplete a word you start typing.  Very quick way to get lines written or to autocomplete a ridiculously long Docker Container ID if you need to get into the folder for something.

Day 6 covers using vim as an editor, and it's something I covered abit before when first starting the posts about Python.  To bring something new on the subject, I am gonna bring up what I've been doing recently to brush up on vim which is a program called vimtutor.  It's a program that to my knowledge comes already included into the distro if it has vim(which most do nowadays by default) and it's essentially a large text document with instructions on how to navigate and use the various vim commands.

Each lesson as you can see above teaches you how to do things in vim, as it's not your standard editor with it's reliance on the different modes + keyboard shortcuts.  Lesson 1.2 early on even teaches what is probably the most googled thing about vi/vim which is "how do you exit vim?" so people can finally leave it!  Best thing is this is repeatable, so working through this once a day has helped me greatly in getting vim down more than just trying to write beginner Python programs in it has.  Definitely look into if you are thinking about starting to use vim.

Github repo pages for these days challenges; Day 4 Day 5 Day 6

One thing that I've been pretty poor at is checking logs.  I think I can count on 1 hand the number of time I've used Event Viewer on Windows in the 21 years I was using it.  It wasn't until I started using NGINX as my reverse proxy that I started actively looking at them.  Browsing the access.log and error.log files gave me a glimpse as to who was accessing my domain and specifically what.  I learned something similar doing the day 3 challenge when it came to /var/log/auth.log.

Day 3 Github Page

Since this is a remote server any login attempt is logged through there, and since I secured the server with keys instead of a password majority of the attempts by the bots don't make it past that part.  There are a few that seem to have a set of keys to use as pictured below, what exactly they are looking for I do not know but I imagine that's the type of thing a honeypot would be good for.

I see a whole bunch of Diffie-Hellman key information in the logs and wasn't sure what they were exactly.  Apparently DH keys are one of the earliest examples of public key protocols and still used by many today, although RSA keys which I am more familiar with came after and are more commonly used as public keys nowadays.  There was a paper also published in October 2015 which I will link to here(don't want to directly link to the PDF download for security reasons, but clicking view publication will download it for those to wish to read it in full) that shows that DH keys are not as secure as believed.  Considering how many unupgraded systems there are on the net it wouldn't surprise me if there are some exposed systems still using this type of keys that the bots are looking to compromise with some keys they found on the darkweb.

Log management is an important role in being a system administrator so just this portion of Day 3 alone was good for me to learn and keep in the back of my mind.

I took a big break during the summer with work starting up again and everything happening with COVID-19 here in the city, I will get back to the Python practice as much as I can during this semester that sees me taking 5 classes while distance learning.

One extra bit of training that I came across while browsing /r/linux was this thing called the [Linux UpSkill Challenge](https://linuxupskillchallenge.com/).  Lovely chap who goes by snori74 on Reddit and Github has a 20 day course setup to help people learn basic remote sysadmin commands.  Importance of people learning how to remotely control and move around their VMs is pretty important with so many services being migrated to the Cloud as opposed to being run locally, and with remote work possibly being more accepted due to this pandemic it's probably even more important.

I know some of these basic commands from working with my Unraid and Proxmox server, plus my normal navigation in Pop!_OS terminal but it doesn't hurt to have it reinforced through practice like this.  And I had $100 in Azure credits through using my student email so it cost me nothing upfront to setup a VPS through them for this.  

Day 0 was to be done anytime before September 7th, 2020 which was simply setting up your VM on whatever service you chose, whether it be DigitalOcean/AWS/Azure/etc.  Creating the VM in Azure gave me the option to download the Private Key, as I opted for enabling SSH through the use of keys as opposed to a password since this was going to be open to the internet.  Now, I never used SSH before on my machine.  So without going too much into specifics of how I setup my SSH I followed this nice [post on Linuxize.com](https://linuxize.com/post/using-the-ssh-config-file/) about how to do it and also how to edit your ssh config file.  As opposed to needing to specific the public IP, port and key needed everytime you can create an alias so instead of needing to type all that, to access my VPS(virtual private server) for this course I simply type

ssh linuxskillup

in the terminal and I am connected to my VPS.  Day 1 is to ensure we can connect to the server and then run some simple commands as shown below.  The day 1 task list on snori74's Github can be found here.

cleverness@Linux-Skillup-Challenge-Ubuntu:~$ ls
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ uptime
  14:56:56 up 3 days, 22:23,  1 user,  load average: 0.00, 0.00, 0.00
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ free
              total        used        free      shared  buff/cache   available
Mem:         938456      188864      170352         908      579240      591652
Swap:             0           0           0
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  1.8G   28G   6% /
devtmpfs        455M     0  455M   0% /dev
tmpfs           459M     0  459M   0% /dev/shm
tmpfs            92M  888K   91M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           459M     0  459M   0% /sys/fs/cgroup
/dev/sda15      105M  9.1M   96M   9% /boot/efi
/dev/loop0       56M   56M     0 100% /snap/core18/1885
/dev/loop1       72M   72M     0 100% /snap/lxd/16740
/dev/loop2       30M   30M     0 100% /snap/snapd/8790
/dev/sdb1       3.9G   16M  3.7G   1% /mnt
/dev/loop3       71M   71M     0 100% /snap/lxd/16922
tmpfs            92M     0   92M   0% /run/user/1000
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ uname -a
Linux Linux-Skillup-Challenge-Ubuntu 5.4.0-1023-azure #23-Ubuntu SMP Mon Aug 17 20:33:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Day 2 was to get used to the concept of how to move across the linux filesystem through the terminal.  Things i've done before but still a good refresher.  In the terminal output below I kept the mistake I made when creating the test directory and not putting the period infront initially to show what happens when you mess up.  Instead of creating the directory locally in my home directory, I accidentally created it in /test under the root directory.  That's why the first ls output after creating the directory was empty.  Removing it and creating it correctly allowed for the output to now be seen as expected.  Day 2 task list can be found here.  Since I am saving time from commuting due to distance learning I am looking to make sure I work on things like this and the Python 100 days of coding more accurately now that I have a sense of how to manage my time between work and school during this unique semester.

cleverness@Linux-Skillup-Challenge-Ubuntu:~$ pwd
/home/cleverness
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ cd /var/log
cleverness@Linux-Skillup-Challenge-Ubuntu:/var/log$ pwd
/var/log
cleverness@Linux-Skillup-Challenge-Ubuntu:/var/log$ cd ..
cleverness@Linux-Skillup-Challenge-Ubuntu:/var$ pwd  
/var
cleverness@Linux-Skillup-Challenge-Ubuntu:/var$ cd ..
cleverness@Linux-Skillup-Challenge-Ubuntu:/$ pwd
/
cleverness@Linux-Skillup-Challenge-Ubuntu:/$ cd /var
cleverness@Linux-Skillup-Challenge-Ubuntu:/var$ cd /log
-bash: cd: /log: No such file or directory
cleverness@Linux-Skillup-Challenge-Ubuntu:/var$ cd ./log 
cleverness@Linux-Skillup-Challenge-Ubuntu:/var/log$ pwd
/var/log
cleverness@Linux-Skillup-Challenge-Ubuntu:/var/log$ ls
alternatives.log  auth.log.1  chrony                 dist-upgrade  dmesg.1.gz  kern.log    lastlog  syslog.1     syslog.4.gz          wtmp
apt               azure       cloud-init-output.log  dmesg         dpkg.log    kern.log.1  private  syslog.2.gz  unattended-upgrades
auth.log          btmp        cloud-init.log         dmesg.0       journal     landscape   syslog   syslog.3.gz  waagent.log
cleverness@Linux-Skillup-Challenge-Ubuntu:/var/log$ ls -l -t
total 1492
-rw-rw-r--  1 root      utmp            292292 Sep  8 08:07 lastlog
-rw-rw-r--  1 root      utmp             12288 Sep  8 08:07 wtmp
-rw-r-----  1 syslog    adm              70469 Sep  8 08:07 syslog
-rw-r-----  1 syslog    adm             252184 Sep  8 08:07 auth.log
-rw-r--r--  1 root      root             87858 Sep  8 07:51 waagent.log
-rw-r--r--  1 root      root             75130 Sep  8 06:11 dpkg.log
drwxr-xr-x  2 root      root              4096 Sep  8 06:11 apt
-rw-r-----  1 syslog    adm               1968 Sep  8 06:11 kern.log
-rw-rw----  1 root      utmp             85632 Sep  8 04:38 btmp
-rw-r-----  1 syslog    adm              36618 Sep  8 00:00 syslog.1
-rw-r-----  1 syslog    adm               4977 Sep  7 00:00 syslog.2.gz
-rw-r-----  1 syslog    adm               3898 Sep  6 00:00 syslog.3.gz
-rw-r-----  1 syslog    adm             256456 Sep  5 23:49 auth.log.1
-rw-r-----  1 syslog    adm              67937 Sep  5 00:00 syslog.4.gz
drwxr-x---  2 root      adm               4096 Sep  4 02:01 unattended-upgrades
-rw-r-----  1 syslog    adm             112130 Sep  3 20:28 kern.log.1
-rw-r--r--  1 root      root              7839 Sep  3 16:33 cloud-init-output.log
-rw-r--r--  1 syslog    adm             260953 Sep  3 16:33 cloud-init.log
-rw-r--r--  1 root      adm              34207 Sep  3 16:33 dmesg
-rw-r--r--  1 root      root              1906 Sep  3 15:23 alternatives.log
-rw-r--r--  1 root      adm              38769 Sep  3 15:22 dmesg.0
drwxr-xr-x  2 landscape landscape         4096 Sep  3 15:20 landscape
-rw-r--r--  1 root      adm              11571 Sep  3 15:18 dmesg.1.gz
drwxr-xr-x  2 root      root              4096 Sep  3 15:18 azure
drwx------  2 root      root              4096 Sep  3 15:18 private
drwxr-sr-x+ 3 root      systemd-journal   4096 Sep  3 15:18 journal
drwxr-xr-x  2 root      root              4096 Aug  3 12:26 dist-upgrade
drwxr-xr-x  2 _chrony   _chrony           4096 May 20 04:16 chrony
cleverness@Linux-Skillup-Challenge-Ubuntu:/var/log$ ls -l
total 1492
-rw-r--r--  1 root      root              1906 Sep  3 15:23 alternatives.log
drwxr-xr-x  2 root      root              4096 Sep  8 06:11 apt
-rw-r-----  1 syslog    adm             252184 Sep  8 08:07 auth.log
-rw-r-----  1 syslog    adm             256456 Sep  5 23:49 auth.log.1
drwxr-xr-x  2 root      root              4096 Sep  3 15:18 azure
-rw-rw----  1 root      utmp             85632 Sep  8 04:38 btmp
drwxr-xr-x  2 _chrony   _chrony           4096 May 20 04:16 chrony
-rw-r--r--  1 root      root              7839 Sep  3 16:33 cloud-init-output.log
-rw-r--r--  1 syslog    adm             260953 Sep  3 16:33 cloud-init.log
drwxr-xr-x  2 root      root              4096 Aug  3 12:26 dist-upgrade
-rw-r--r--  1 root      adm              34207 Sep  3 16:33 dmesg
-rw-r--r--  1 root      adm              38769 Sep  3 15:22 dmesg.0
-rw-r--r--  1 root      adm              11571 Sep  3 15:18 dmesg.1.gz
-rw-r--r--  1 root      root             75130 Sep  8 06:11 dpkg.log
drwxr-sr-x+ 3 root      systemd-journal   4096 Sep  3 15:18 journal
-rw-r-----  1 syslog    adm               1968 Sep  8 06:11 kern.log
-rw-r-----  1 syslog    adm             112130 Sep  3 20:28 kern.log.1
drwxr-xr-x  2 landscape landscape         4096 Sep  3 15:20 landscape
-rw-rw-r--  1 root      utmp            292292 Sep  8 08:07 lastlog
drwx------  2 root      root              4096 Sep  3 15:18 private
-rw-r-----  1 syslog    adm              70469 Sep  8 08:07 syslog
-rw-r-----  1 syslog    adm              36618 Sep  8 00:00 syslog.1
-rw-r-----  1 syslog    adm               4977 Sep  7 00:00 syslog.2.gz
-rw-r-----  1 syslog    adm               3898 Sep  6 00:00 syslog.3.gz
-rw-r-----  1 syslog    adm              67937 Sep  5 00:00 syslog.4.gz
drwxr-x---  2 root      adm               4096 Sep  4 02:01 unattended-upgrades
-rw-r--r--  1 root      root             87858 Sep  8 07:51 waagent.log
-rw-rw-r--  1 root      utmp             12288 Sep  8 08:07 wtmp
cleverness@Linux-Skillup-Challenge-Ubuntu:/var/log$ ls -l -t -r -a
total 1500
drwxr-xr-x   2 _chrony   _chrony           4096 May 20 04:16 chrony
drwxr-xr-x   2 root      root              4096 Aug  3 12:26 dist-upgrade
drwxr-xr-x  13 root      root              4096 Aug 14 17:41 ..
drwxr-sr-x+  3 root      systemd-journal   4096 Sep  3 15:18 journal
drwx------   2 root      root              4096 Sep  3 15:18 private
drwxr-xr-x   2 root      root              4096 Sep  3 15:18 azure
-rw-r--r--   1 root      adm              11571 Sep  3 15:18 dmesg.1.gz
drwxr-xr-x   2 landscape landscape         4096 Sep  3 15:20 landscape
-rw-r--r--   1 root      adm              38769 Sep  3 15:22 dmesg.0
-rw-r--r--   1 root      root              1906 Sep  3 15:23 alternatives.log
-rw-r--r--   1 root      adm              34207 Sep  3 16:33 dmesg
-rw-r--r--   1 syslog    adm             260953 Sep  3 16:33 cloud-init.log
-rw-r--r--   1 root      root              7839 Sep  3 16:33 cloud-init-output.log
-rw-r-----   1 syslog    adm             112130 Sep  3 20:28 kern.log.1
drwxr-x---   2 root      adm               4096 Sep  4 02:01 unattended-upgrades
-rw-r-----   1 syslog    adm              67937 Sep  5 00:00 syslog.4.gz
-rw-r-----   1 syslog    adm             256456 Sep  5 23:49 auth.log.1
-rw-r-----   1 syslog    adm               3898 Sep  6 00:00 syslog.3.gz
-rw-r-----   1 syslog    adm               4977 Sep  7 00:00 syslog.2.gz
-rw-r-----   1 syslog    adm              36618 Sep  8 00:00 syslog.1
drwxrwxr-x  10 root      syslog            4096 Sep  8 00:00 .
-rw-rw----   1 root      utmp             85632 Sep  8 04:38 btmp
-rw-r-----   1 syslog    adm               1968 Sep  8 06:11 kern.log
drwxr-xr-x   2 root      root              4096 Sep  8 06:11 apt
-rw-r--r--   1 root      root             75130 Sep  8 06:11 dpkg.log
-rw-r--r--   1 root      root             87858 Sep  8 07:51 waagent.log
-rw-r-----   1 syslog    adm             252184 Sep  8 08:07 auth.log
-rw-r-----   1 syslog    adm              70469 Sep  8 08:07 syslog
-rw-rw-r--   1 root      utmp             12288 Sep  8 08:07 wtmp
-rw-rw-r--   1 root      utmp            292292 Sep  8 08:07 lastlog
cleverness@Linux-Skillup-Challenge-Ubuntu:/var/log$ cd
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ ls -ltra
total 36
-rw-r--r-- 1 cleverness cleverness  807 Feb 25  2020 .profile
-rw-r--r-- 1 cleverness cleverness 3771 Feb 25  2020 .bashrc
-rw-r--r-- 1 cleverness cleverness  220 Feb 25  2020 .bash_logout
drwxr-xr-x 3 root       root       4096 Sep  3 15:18 ..
drwx------ 2 cleverness cleverness 4096 Sep  3 15:18 .ssh
drwx------ 2 cleverness cleverness 4096 Sep  3 15:20 .cache
-rw-r--r-- 1 cleverness cleverness    0 Sep  3 15:21 .sudo_as_admin_successful
drwx------ 3 cleverness cleverness 4096 Sep  3 16:34 .config
drwxr-xr-x 5 cleverness cleverness 4096 Sep  3 16:34 .
-rw------- 1 cleverness cleverness  162 Sep  7 15:25 .bash_history
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ man mkdir
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ pwd
/home/cleverness
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ mkdir /test
mkdir: cannot create directory ‘/test’: Permission denied
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ sudo mkdir /test
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ ls   
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ sudo rm -rf /test
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ mkdir ./test
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ ls
test
cleverness@Linux-Skillup-Challenge-Ubuntu:~$ cd ./test
cleverness@Linux-Skillup-Challenge-Ubuntu:~/test$ 

While surfing Reddit I came across someone who had this application listed in their docker apps, I googled it and instantly wanted it for my Plex server.  Essentially allows other users that have access to your Plex server to search for content.  If the content is already available it will show as such and even give a "View on Plex" link which opens Plex in another tab to the content(at the moment this works on majority of my library, but some links are broken), or if its not available they can request the media.  It works with Jellyfin/Emby as well but at the moment I run Plex, so this seems like a efficient way to get family and friends to request stuff if I happen to be busy without me forgetting about it later when I get home.

The author has the main code hosted on his github which can be found quickly off his website at https://ombi.io/ but I went with the linuxserver.io image to put on my Unraid server.  Installation is pretty simple, and since I'm using Plex it let me log in with my Plex credentials and populated my Plex token and IP address, although since I run Plex in a container the port had to manually be changed to connect properly.

There's an option to import the user list from the Plex server if you are the admin there, and after that they can login with just their Plex credentials if you enable Plex OAuth in the settings.  They have a mobile app on the Google Play Store which I ended up downloading since I wanted all request notifications to go through on my phone as a push notification.  I think the app and admin access together cost 4-5 bucks but so far it seems to be worth it for what I need.  After logging in on mobile my username eventually ended up on the mobile devices registered list under Settings > Notifications > Mobile Notifications and I was able to setup notifications, so now all requests will ping my phone.  

Email took me abit because I am dumb.  I made a gmail account for server email use and tried to register that and kept getting failed connection errors in the log. Double checked the google approved smtp ports and made sure IMAP was enabled on the gmail account and still an error.  Of course the problem was a simple solution; I only ever opened the management port when launching the docker image, needed to relaunch it again with an additional port for the smtp email.  After that it worked fine, test email and mass email/newsletter both worked.

There is sonarr/radarr/couchpotato integration here to automatically download stuff that I approve it seems but I haven't gone that far with this yet, only got it going today.  After seeing how it goes over a period of time I'll probably look into that storage space permitting.  App looks awesome so far though.

One step of my second assignment in constructing a playlist involved reading in a list of csv files to construct the playlist from.  I wanted to make a pretty comprehensive way to do it without listing all the files in the program incase we had to revisit this for the final assignment, or if I just wanted to reuse it for something else later on.  So I went to look at the java docs for the File class we usually work with to see all the options, which you can see here.

Looking at all the methods the most useful one is the listOfFiles method, which returns an array of all the file names with their relative paths.  Assuming all files we want read are being placed in a folder in the project directory called 'csvfiles', an example of this in use can be seen below.

 File folder = new File("csvfiles");
 File[] listOfFiles = folder.listFiles();

First line creates an object called folder of type File, and it's referencing the relative path of the csvfiles folder.  Second line creates an array of objects called listofFiles of type File, and the listFiles method will return all the files in the folder and save it to the array.  But let's say you want to either output all the files that are being read, or want to verify that the files are being read correctly.

int numFiles = 0;
for (File file : listOfFiles) {
	if (file.isFile()) {
    	System.out.println(file);
        numFiles++;//counts number of files in directory
    }
}

We create an object file of type File, that will iterate as long as the array listOfFiles has values in it(in my case it will go 5 times).  The isFile() method is essentially a boolean check, if it reads a proper file path it will execute, if not then it skips the if loop.  We print out the file path and then with a counter variable created before the for loop we count the number of files total that are being read.  Sample output for just this step should look something like this

csvfiles\regional-global-weekly-2020-01-17--2020-01-24.csv
csvfiles\regional-global-weekly-2020-01-24--2020-01-31.csv
csvfiles\regional-global-weekly-2020-01-31--2020-02-07.csv
csvfiles\regional-global-weekly-2020-02-07--2020-02-14.csv
csvfiles\regional-global-weekly-2020-02-14--2020-02-21.csv

I'm using the same sets of data that I talked about in my last post but including more than just the 1-17 to 1-24 sets of streaming data from Spotify.  From here I've validated the listofFiles array is valid and I can either use that to create a Queue for each one to later merge , or I can create a large Queue whose total size is that of all the files times the number of datasets per file which in the case of spotify csv's are 200.

int size = 200;//Total number of songs in a spotify csv file
int trueSize = size * numFiles;//equal to 1000

The counter numFiles is not necessary depending on how you are looking to execute the reading of all the files into Queues, but it's an option and one to keep in mind depending on exactly what type of data you are looking to read and parse.

Part of my degree path in Information Systems requires taking my programming classes in Java so far, and that includes Data Structures.  Due to the nature of the course, majority of our assignments involve parsing data in a manner that we judge appropriate relative to the data we are working with and the task assigned.  The first 2 assignments involved taking the Top 200 Streaming Songs list off Spotify and making first just a simple linked list that prints out, and then a playlist.

Spotifycharts.com makes it easy to export their data into a csv spreadsheet, there is a lovely button in the top right corner.  The challenge that came with this first assignment is that a CSV file is short for comma-seperated values.  If you download the CSV for lets say the week of 1-17-2020 to 1-24-2020 and open it, Windows defaults CSV files to excel and the data is nice and organize with no commas present.  Right click and open it in Notepad and you'll see the actual raw data and see the column values seperated by commas on every line.  So this essentially is where we had to decide how we wanted to read the data.  Rather than try and fiddle with excel as I am not fond of it, I chose to just read the CSV data raw.

Referencing the official java docs on the String class there is a method of the string class called split, where you can specify the regular expression you want it split by.  For CSV the choice is obviously a comma. For normal pieces of data this would normally be enough and you might not need anything else to parse a CSV file, but due to the nature of music extra commas either in song titles or including other featured artists on the track aren't uncommon.  A snippet of code of my first version of our first assignment in the main method.

        TopStreamingArtists artistNames = new TopStreamingArtists();
        String csvFile = "src/regional-us-weekly-2020-01-17--2020-01-24.csv";
        String inputLine;

        try {
            Scanner inputStream = new Scanner(new FileReader(csvFile));
            inputLine = inputStream.nextLine();
            while (inputStream.hasNextLine()) {
                inputLine = inputStream.nextLine();
                String[] tempArray = inputLine.split(",");
                int tempPosition = Integer.parseInt(tempArray[0]);
                String tempTrack = tempArray[1];
                String tempArtist = tempArray[2];
                int tempStreams = Integer.parseInt(tempArray[3]);
                String tempUrl = tempArray[4];

                artistNames.insertLast(tempPosition, tempTrack, tempArtist, tempStreams, tempUrl);
            }
            inputStream.close();

So the goal was to take my file and read it with a Scanner object I created, and read it one line at a time and save that line as a string in the variable inputLine, and then split it into an array of type String called tempArray at every comma so that I'd have abunch of subStrings of that read line that I can then place in temp variables, and add to the linked list.  First issue that came up is an error because the very top of the file has 2 lines that aren't necessary data we want to parse, just data relevant to reading the file if done so with the human eye.  This can be solved by adding 2 "inputLine = inputStream.nextLine();" lines of code before executing the while loop but it's not really effective if you try to reuse this program later for some other pieces of data.  What if you get a csv file with 30 lines of text before the data even appears, adding "inputLine = inputStream.nextLine();" 30 times is super messy.

If you ignore that or modify the CSV file to take those out, the second issue that came up around 85 lines in is that this code will literally separate at every comma as it's supposed to.  So when the program got to Eminem's Yah Yah (feat. Royce Da 5'9", Black Thought, Q-Tip & Denaun), as you can see there are 2 extra commas in the track field alone.  So the tempArray has more values in it that the previous lines did, and the positions will be all off, so then trying to ParseInt the 4th value in the array, instead of reading what should be the number of streams it will be reading Q-Tip & Denaun and throw an error as this cannot be made into an integer value.  The first assignment was flexible and we could manipulate the CSV file so I just submitted it with only the first 65 lines for grading.  But considering everything else builds on this I wanted to make sure the second assignment was done properly, as it's literally using the same sets of data we used for the first one.

As our professor said either before or after the first assignment(I don't remember at this point, was roughly 2 months ago), there's no point in trying to reinvent the wheel in regards to doing something like reading CSV files.  It's been something that has been down for a LONG time at this point and there are many available libraries that we can use and import into our java projects to assist with that.  Some time spent googling and I came open OpenCSV.  I downloaded the opencsv-5.1.jar file to get the library and it requires the Apache library as a dependency so I downloaded commons-lang3-3.9.jar.  IntelliJ has a GUI for importing libraries but follow your IDE documentation for doing so if its different.  After doing that I made a separate program of my first assignment that would utilize OpenCSV and imported the following 2 libraries up top to get the library loaded

import com.opencsv.CSVReader;
import com.opencsv.exceptions.CsvValidationException;

So to see how that changes the code here's a snippet of the updated version for the first assignment

        SortedArtists artistNames = new SortedArtists();
        String csvFile = "src/regional-us-weekly-2020-01-17--2020-01-24.csv";

     try {

            CSVReader reader = new CSVReader(new FileReader(csvFile));
            String[] inputLine = reader.readNext();
            reader.skip(2);                    

            while ((inputLine = reader.readNext()) != null) {

                int tempPosition = Integer.parseInt(inputLine[0]);
                String tempTrack = inputLine[1];
                String tempArtist = inputLine[2];
                int tempStreams = Integer.parseInt(inputLine[3]);
                String tempUrl = inputLine[4];

                artistNames.insertLast(tempPosition, tempTrack, tempArtist, tempStreams, tempUrl);
            }
            reader.close();//stops reading file after while loop

CSVReader is a class in OpenCSV that is primarily used for this, **full java doc here**to see all the methods.  First important addition it added was the skip method, because by default it won't skip any lines like the first version and will read from the beginning, which will give the same error.  By using the skip method and specifying the number of lines to skip, it will automatically do so and it keeps the code clean.  I know that spotifycharts.com will always have the first 2 lines populated with information that I don't want to read, but if I wanted to reuse this code somewhere else and it needs to be modified I could change the 2 to whatever number was required, change it to a variable, even add in a user prompt to ask how many lines to skip if the program is going to be a multipurpose CSVReader program and save it to a variable and pass that into the skip method.

The second important addition which isn't immediately clear unless you open and read the CSVReader class and the readNext() method is it does all the parsing of the line and even removes quotes by default.  It does all the proper parsing to account for things like extra commas and will split the line correctly, so running this version of the program I get a much cleaner output when displaying the linked list.  Doing so made parsing data for the second assignment easier since it was essentially the same data sets.

Going forward I essentially learned to look for a library that does whatever is giving me trouble, as when it comes to Data Structures someone else probably had to deal with the same issue before and there's a library that takes care of it.  Both versions of this program can be seen on my personal git repository for comparison right here.  Master version is what I submitted first, and version 2 branch is the updated one that uses OpenCSV to parse.  The included CSV file was the edited one that doesn't have the first 2 lines so there is no reader.skip(2) code in version2 branch  as I had placed here as an example, but that was used in the following program.

When first making my Proxmox server I just used drives I had left over from upgrading previous builds, I had the boot drives as a set of 120gb Sandisk SSDs, 2 Crucial 250gb SSDs as a ZFS Raid-1 holding my VMs, and a 250gb Corsair Force MP510 setup as a Directory running backups because I thought it would help the speeds when backing up my VMs(it really didn't).  As using an NVMe drive in the onboard m.2 slot on the Gigabyte B450M Aorus motherboard limits the number of SATA ports to 4 I found myself running low on disks, so the plan is to migrate the boot mirror(which Proxmox calls rpool) to the 2 Crucial SSDs and run most of the VMs directly off that.

The Proxmox official instructions for replacing a failed disk in the root pool covered majority of the process, but I had to take a few extra steps as I was upgrading the size of the pool. Once I made sure all the VMs were backed up on the nvme I pulled one of the Sandisks and opened up the shell in proxmox.  Since both of my boot drives were currently fine the pulled Sandisk served as my failsafe incase I messed anything up in the process, as it wouldn't be touched until the process was complete.

  root@einherjar:~# zpool status -v
  pool: rpool
  state: ONLINE
  scan: resilvered 63.5G in 0 days 00:27:32 with 0 errors on Sat Apr  4 09:24:53 2020
  config:
  NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdd3    UNAVAILABLE  0     0     0
        sdc3    ONLINE       0     0     0

Whatever disk you pull is going to show up unavailable.  They show up here as partitions /dev/sdd3 and /dev/sdc3 but disk id's can also show up, just go off whatever your pool status shows.  After identifying the pulled drive it needs to be put offline in the pool, run status again to confirm.

  root@einherjar:~# zpool offline rpool /dev/sdd3
  root@einherjar:~# zpool status -v
  pool: rpool
  state: DEGRADED
  status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
  action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: resilvered 63.5G in 0 days 00:27:32 with 0 errors on Sat Apr  4     09:24:53 2020
  config:
  NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdd3    OFFLINE      0     0     0
        sdc3    ONLINE       0     0     0

From here plug the replacement drive in(my Crucial 250 in this case) and get the disk location either in the Proxmox GUI or by running fdisk -l in the shell(we'll go with /dev/sdb for the guide)

Next step was to copy the partition table over from the current Sandisk 120gb SSD to the crucial.  As the official wiki stresses, the order the drives are placed in the sgdisk command are important as doing it wrong will wipe your current boot disk and that pool, taking all the proxmox configs with it.

sgdisk --replicate=/dev/target /dev/source

So the target disk goes first, and the source second.  In the shell in my case it goes

  root@einherjar:~# sgdisk --replicate=/dev/sdb /dev/sdc

This should copy the partitions over.  Now at this step if you are like me and upgrading the disks to bigger size, you will need to resize the partitions of the drive(which in my case in Proxmox is sdb3) as its copying over a 120gb partition over and that means we have roughly 130gb of free space on this drive that won't be used otherwise.  I took the wikis advice and used parted to resize it, which I had to install.

Run parted first on your current root disk to see something similar to the following

root@einherjar:~# parted /dev/sdc
GNU Parted 3.2
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: ATA CT250MX500SSD1 (scsi)
Disk /dev/sdc: 250GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      17.4kB  1049kB  1031kB                     bios_grub
 2      1049kB  538MB   537MB   fat32              boot, esp
 3      538MB   249GB   248GB   zfs

Of note is the disk size in line 7, and the end sector for partition 3 in line 15 as that is the partition we are going to resize.  Run parted or fdisk on your new disk to verify the disk size and then run the following commands

root@einherjar:~# parted /dev/sdb
GNU Parted 3.2
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) resizepart                                                            
Partition number? 3

Then specify the size, which for me I input 248GB but in actuality it should have been 249GB, as looking at the free space available on the disk in fdisk I left 1gb out.  So now the zfs pool partition on the disk is the correct size.

root@einherjar:~# sgdisk --randomize-guids /dev/sdb
The operation has completed successfully.

Wiki suggests to do this to avoid confusion in the server, and I didn't want to deal with a kernel panic so there it goes.  And next step which didn't work for me

grub-install /dev/sdb

If this completes for you, great.  If it doesn't, you need to fix this otherwise the drive won't be able to boot the OS as I came to find out.  Easiest way I got this going was to just copy the working bios_grub and EFI partitions off the currently working boot drive, which are partitions 1 and 2.  Thankfully after running sgdisk command the new drive has the same partitions in the same start sectors

# dd if=/dev/sdc1  of=/dev/sdb1 

The above command tells dd to use /dev/sdc1 as input file and write it to output file /dev/sdb1.  Do this again for partition 2, and it should be able to properly boot now.

We can now replace the pulled sandisk with the crucial drive in the rpool.  Syntax for the command is

zpool replace <pool> <device> [new-device]

Where first device is the name of the drive you pulled, and new device is the one we've been formatting all this time.  Again you need to write down the device exactly as it shows up in zpool status -v, so if the pool is listing disk-id's or UUIDs for either drive we are swapping go with that or it won't work.

root@einherjar:~# zpool replace rpool /dev/sdd3 /dev/sdb3
Make sure to wait until resilver is done before rebooting.

You might need a -f flag at the end of the replace line, if the shell prompts for it then add it in as I needed to do for this first swap.  This will start the resilvering process where the data from the current root drive gets mirrored over to the new root drive.  Do not power down the server until this is done.  Check on the status and you should see a similar output to below depending on your drives/pool size

  root@einherjar:~# zpool status -v
  pool: rpool
  state: DEGRADED
  status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
  action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Apr 4 09:27:53 2020
  	27G scanned out of 64G at 4.46M/s, 10m to go
    26G resilvered, 40.60% done
  config:
  NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
	    sdc3         ONLINE       0     0     0
	    replacing-1  OFFLINE      0     0     0
	      sdd3       OFFLINE      0     0     0
	      sdb3       ONLINE       0     0     0  (resilvering)

When its finally done should see something similar to

root@einherjar:~# zpool status -v
  pool: rpool
 state: ONLINE
  scan: resilvered 63.5G in 0 days 00:27:32 with 0 errors on Sat Apr  4 09:37:53 2020
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb3    ONLINE       0     0     0
            sdc3    ONLINE       0     0     0

Power the system off and pull the old root drive and power it on to make sure the new drive boots and all your server settings are the same.  If it did then great, next step is to repeat everything all over again with the 2nd new drive to copy everything over and resilver the pool again.  This will give you a new boot pool with no errors in the zpool.

Last step is to turn on autoexpand for the pool, otherwise in my case I am stuck with 120gb available for use with VMs when the disks are obviously much bigger than that now.  Running zpool list will confirm whats actually available for use.  A very simple

root@einherjar:~# zpool set autoexpand=on rpool

As rpool is the name for the boot pool in Proxmox will turn autoexpand on(which when checking before was default to off) and running zpool list should now show a pool size almost equal to the size of your drives(barring a few MB for the boot and EFI partition)

root@einherjar:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool   231G  87.4G   144G        -         -     2%    37%  1.00x    ONLINE 

With that my boot pool has now been upgraded and I have some more flexibility in my setup.  After migrating the VMs over from the NVMe drive to the local-zfs pool on the boot drives I used fdisk to nuke it and rebooted it so changes would take effect, and then created a LVM storage on it to run VMs that can take advantage of the higher speeds of the drive, or ones that would be negatively affected by other VMs on the local-zfs storage causing an I/O delay.  Anything like a Minecraft server or music/video server that is handling things in real time can suffer delays if even SSDs are busy with things like scheduled backups or heavy writes from 1 or more of the VMs running on it.  Moving it to a separate drive will reduce that and most game servers in particular benefit from running off an NVMe drive when its available.

I replaced one of the Sandisks with a 1TB Hard Drive I pulled from a PS4 Pro when I upgraded it to a 2TB drive and made a new Directory called Backup with that, and even though its a 5400RPM drive I was still getting a constant 160MB in read/writes when backing up from the SSDs, and alittle faster when backing up from the VMs on the NVMe.  Rebuilding is abit slower from it but since this is for home use it's a good trade-off in exchange for more space to hold backups.  The 4th slot in my drive cage is just occupied by a sandisk I formatted empty for airflow purposes, but I already plan to replace it with another SSD later on when I take on my next hardware upgrade for the Proxmox server in trying to setup a separate Windows VM with GPU passthrough to play certain games I won't have access to when I migrate my desktop OS to Linux again.

I had an original boot disk and backups of all my VMs on a separate Directory so this was a good learning experience in managing zpools without much fear of losing everything.  Since Proxmox is essentially Debian at its core the process carries over to other Linux distros that support ZFS so it's at least something that I can call back on going forward.