Home | Zones | Log in | Register
Users | Join | Wiki

Servers: Automation, Homelabs, Networking, Storage, Hardware, & More

cj docker filesystems home server plex servers

Nobody
From: hey ✪ | Posted: 2024-01-04 04:22:102024-01-04 04:22 | Filter

SERVER CJ: Lots and lots of servers!

Talk about servers here. Home servers, work servers, custom build servers, OEM pre built servers, networking, file systems, containers, virtualization, and all the cool stuff you are doing with your computers for work and play.

Servers you love, servers you hate, servers you want to build, servers you want to try out, if its a computer and you do stuff with it, talk about it here!

OLD SERVERS!

old-server-1766511755.jpg

NEW SERVERS!

Dell-PowerEdge-servers-generic-2328069588.jpg

BIG SERVERS!

iStock_24458617_LARGE-2361512260.jpg

SMALL SERVERS!

Raspberry-Pi-Zero-USB-hub-3509520560.jpg

WEIRD SERVERS!

26j1-lF0TSD4_JOqibcqqDkh3YowaAhb84ZAPg.png

SERVERS FROM AROUND THE WORLD!

colorful-servers-around-globe-23277589-3795775377.jpg

Cloud Server

Dont want a physical server living in your house? Consider using the cloud! You can spin up a virtual private server (VPS) easily with providers such as AWS and Digital Ocean, among others. Cloud is a great option for applications that need high-availability.

Home Media Server

Want to build a home media server but dont know where to start? Do you love Linux? Well maybe the “Perfect Media Server” will be a good choice for you!

More resources:

Building a Plex Server

What is the Best CPU for Plex?

tl;dr: get a 7th gen or later Intel CPU if possible in order to utilize Quick Sync (used for transcoding e.g. down-converting media to play on mobile devices, etc.). You can use older CPU’s just fine but they will not have as advanced Quick Sync capabilities. If you do not transcode and only play the source media directly as-is, then you should be fine with even less.

While not needed for Plex, if you do wish to encode HEVC 10-bit, then you should look at Intel 11th generation or later. The integrated graphics on those processors are capable of encoding HEVC 10-bit.

Regardless of your requirements, I suggest going with an Intel 7th generation or later, to get the best experience out of your Plex server.

Networking

Data Storage

resources:

Building a NAS Server

The best guide around for general purpose building of a basic NAS (network attached storage) server

Object Storage

An alternative to traditional filesystems, object storage can be run at home (or in the cloud) using services such as:

Data Sharing

SMB Share

So you have your storage on one server (or PC, etc.), and you want to access it from another server (or PC, etc.), on the same network. How do you do that? You can do this with an SMB share.

Resources

Create SMB Share

Different operating systems will have different methods for creating a SMB share. For example, on macOS you can use the System Preferences > Sharing > File Sharing menu options to create and configure a SMB share (you can also do it from the command line; link). Note that you may or may not also want to set up guest access to the share, and or create a share user just for sharing.

Create SMB Share on Linux

Connect to SMB Share

Different operating systems will have different methods for accessing a SMB share.

On Ubuntu 22.04, you first need to install some extra tools

sudo apt install samba cifs-utils smbclient

You can check if a SMB share is available on the network with a command like

smbclient -L 192.168.1.10 -U guest

Where the IP is the address of the computer on your network hosting the share, and “guest” is used as the user assuming you enabled guest access to the share from the share host. If it works, you should see a list of available share points.

You can manually mount a SMB share with a command like this

# need to create the mount point first
mkdir -p /media/share/Media

# try to mount the share
sudo mount -t cifs -o guest,username=guest,vers=3.0,uid=yourlocalusername //192.168.1.10/Media /media/share/Media

In this command;

//192.168.1.10/Media is the local network address to the SMB share “Media” on the system at IP address 192.168.1.10. Note that the share point name “Media” does not include the path to the directory on the local system in the SMB address.

we are using both the guest mounting option to disable the password prompt, and the username=guest to mount it under the remote guest user access available on the remote SMB host ; you might want to exclude this if you are not using guest access on the SMB share

vers=3.0 is specifying to use SMB Version 3.0 ; there have been numerous security vulnerabilities found in old versions of SMB, so this option forces the local client to try and connect to the remote host with one of the most modern versions of SMB which is currently 3.0

uid=yourlocalusername, where yourlocalusername can be replaced with the username of your current user, changes the file ownership of the mounted files on the local system to that of the current user, in order to alleviate some issues with file permissions when trying to write to the files and directories on the SMB share

/media/share/Media is the path to the location on the local system where the SMB should be mounted; make sure this directory path already exists, and you may need to adjust the ownership of that location for your current user as well (cant remember if I needed to do that or not but I usually end up having to do it for other mount points with sudo chown -R myusername /path/to/dir)

If it does not work, you may need to check the system logs with

sudo dmesg

If it does work, you should not see any error messages, and you should be able to see the contents of the SMB share at the specified local path with ls -l.

Once you get it working, you can unmount the share (sudo umount /media/share/Media) and then add an entry to your /etc/fstab, it might look like this

# open the fstab file; always a good idea to make a backup copy first!
sudo nano /etc/fstab
...

# edit the file; add a line like this to the file:
//192.168.1.10/Media /media/share/Media cifs    guest,username=guest,vers=3.0,uid=yourlocalusername     0       0

# save and close the file

# reload all the mount points from your /etc/fstab
sudo mount -a

If you have issues, troubleshoot based on error messages from dmesg, and keep in mind that you can copy / paste most error messages into Google and even ChatGPT to search for some guidance.

Note also that you may need to allow SMB access through your firewall, if you have it enabled, with a command such as

sudo ufw allow Samba

This requires that you have installed the samba package from apt. You can check your firewall configuration with commands like

sudo ufw status
sudo ufw app list

Resources

Containers

Lighter than a virtual machine, heavier than a virtual environment, containers are a popular way to manage applications in an isolated manner that keeps apps’ software installations from colliding with each other, and with the host operating system. This allows for greater portablity, reproducibility, and orchestration of applications on your system. Many types of containers can be shared freely on remote container repositories such as Docker Hub and Quay.io, and containers can usually be created in a scripted manner using a recipe file which lists all the commands needed to install and configure desired packages on top of a base operating system image.

Docker

Docker is the de facto standard for portable containers. Note that the term “Docker” can refer both to the container format, and to the engine which runs the containers; some other non-Docker container engines are compatible with Docker containers. Many containers are freely available on Docker Hub, and Docker containers can be (re)built using the Dockerfile recipe format. Multiple independent containers can be orchestrated at the same time with the Docker Compose plugin.

Resources

How to install Docker

IMPORTANT NOTE:
If you use ufw or firewalld to manage firewall settings, be aware that when you expose container ports using Docker, these ports bypass your firewall rules. For more information, refer to Docker and ufw.

Official docs here; https://docs.docker.com/engine/install/ubuntu/

  • Assuming you are on Ubuntu 20.04 / 22.04 and you do not have Docker already installed

    # Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# install the latest version of Docker
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# verify that its running
sudo docker run hello-world

If you are having trouble getting Docker to run, you can do a couple things to investigate;

# try to start the docker daemon
dockerd

# check the status of the Docker service
systemctl status docker

If Docker does not start automatically on system boot, you can also configure that like this;

sudo systemctl enable docker.service
sudo systemctl enable containerd.service

DOCKER COMPOSE

Networking

How to set a static IP address for a device on your local network

How to set up port forwarding

  • pfsense:


How to get your current IP address on your internal local network

(Linux, macOS)

ifconfig

will show the details for all network devices, amongst them should be some devices with names like eth0, with IP addresses such as 192.168.x.x

Windows: https://www.howtogeek.com/858334/how-to-find-your-ip-address-from-cmd-command-prompt/

How to get your public internet IP address

curl http://checkip.amazonaws.com
# or
curl ifconfig.me

How to get IP address(es) of website

nslookup github.com
# or
dig github.com

How to check that a server (local or remote) can be reached

ping 8.8.8.8

How to check what local process is running on a port

sudo lsof -i :1234

How to check which local ports are in use

lsof -i TCP | grep LISTEN

How to check if a port is open on a server

# netcat
nc -vz 192.168.1.2 7878
# Connection to 192.168.1.2 7878 port [tcp/*] succeeded!

nc -vz 192.168.1.2 7879
# nc: connect to 192.168.1.2 port 7879 (tcp) failed: Connection refused

Piracy

Arrrrr see the usenet thread here COMING SOON

Linux

Linux thread here COMING SOON

❧ Edited by hey at 2024-01-04 04:43:262024-01-04 04:43
➕
From: hey ✪ | Posted: 2024-01-04 04:35:492024-01-04 04:35 | Filter

Docker Compose for *arr apps

Docker Compose is a plugin for Docker that lets you use a YAML file to define the config for multiple containers, and start and stop them all at once (or individually) as needed.

resources:

Make sure you installed compose when you installed Docker

sudo apt-get install docker-compose-plugin
# verify that its installed
docker compose version

Each of the Wiki pages for the -*arr apps includes an example Docker Compose YAML. You can use a single YAML file with all of the services listed for easier management. See the link there for the guide’s example YAML configuration.

in this example, the contents of a file named docker-compose.yml look like this:

---
version: "2.1"
services:
  radarr:
    image: lscr.io/linuxserver/radarr:latest
    container_name: radarr
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=EST
    volumes:
      - /home/username/radarr:/config
      - /media/share/Media:/media/share/Media
    ports:
      - 7878:7878
    restart: unless-stopped
  bazarr:
    image: lscr.io/linuxserver/bazarr:latest
    container_name: bazarr
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=EST
    volumes:
      - /home/username/bazarr:/config
      - /media/share/Media:/media/share/Media
    ports:
      - 6767:6767
    restart: unless-stopped
  lidarr:
    image: lscr.io/linuxserver/lidarr:latest
    container_name: lidarr
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=EST
    volumes:
      - /home/username/lidarr:/config
      - /media/share/Media:/media/share/Media
    ports:
      - 8686:8686
    restart: unless-stopped
  sonarr:
    image: lscr.io/linuxserver/sonarr:latest
    container_name: sonarr
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=EST
    volumes:
      - /home/username/sonarr:/config
      - /media/share/Media:/media/share/Media
    ports:
      - 8989:8989
    restart: unless-stopped
  • the PUID and PGID values match the values given by the id command for my local server user account
  • I have put the config dir for each service under my local home dir, but mapped it to /config inside the container, which is the default location inside the container for configs
  • I mapped the local volume /media/share/Media to the same address inside the container, for all containers, for consistency (optional)
  • changed all timezones to EST
  • remember that Docker ports bypass the firewall (ufw) so these ports will be accessible on the local network

Start the containers with the command

docker compose up -d

This starts all containers and runs them in a detached state in the background.

Check on which containers are running with

docker compose ps

(you can also use docker ps but its less descriptive)

Stop all the containers with

docker compose down

You can also start / stop / restart containers individually

docker compose up lidarr -d

Once the containers are running, you can navigate to your web browser to access them. Navigate to a URL that looks like http://192.168.1.123:8989/, inserting the local IP address of the server running the services.

See the guides at LinuxServerIO and Servarr Wiki and TRaSH Guides and the official docs for each service for further setup instructions for each service.

➕
From: hey ✪ | Posted: 2024-01-04 04:39:402024-01-04 04:39 | Filter

Interacting with FTP Server

lftp

You can use the command line tool lftp to interact with FTP servers in order upload and download files, and even mirror entire directories.

lftp home page is here; https://lftp.yar.ru/

To install lftp, on macOS you can use Homebrew (brew install lftp), and on Ubuntu you can use the package manager (sudo apt install lftp). Once installed, you can see the tool’s internal documentation with man lftp.

Check the man pages for lftp to see the full details of commands, settings, and more.

lftp can be used interactively, or non-interactively, from the command line. In interactive mode, lftp will start a new session inside your terminal connecting you to the remote FTP server. From there you can use a handful of commands to browse the FTP file system and upload / download files and directories, among other things.

You can connect to an FTP server with a URL in a format like this:

ftp://username:password@server.com

where username and password are the username and password you plan to log in with; you can omit the password and enter it at the interactive prompt if you like, though it might be a good idea to keep the username in the URL so it does not try to automatically log you in as a user you did not intend for. If the username or password contains special characters that your shell does not like, you may need to wrap some or all of the URL in single (‘) or double (“) quotes.

The full command should look like this

lftp ftp://username:password@server.com

and if it works you should be presented with an interactive prompt

lftp username@server.com:~> pwd
ftp://username:password@server.com

lftp username@server.com:~> ls
drwxrwxrwx   1 user     group                0 Jan 01  1970 ..
drwxrwxrwx   1 user     group                0 Jan 01  1970 .
drwxrwxrwx   1 user     group                1 Dec 07 20:52 Books
drwxrwxrwx   1 user     group                1 Nov 27 22:01 Movies
drwxrwxrwx   1 user     group                1 Dec 07 18:04 TV
lftp username@server.com:/>

lftp has its own set of commands that mimic (but are NOT 100% equivalent to) your standard POSIX shell commands, such as ls, pwd, and cd.

Importantly, unlike a normal shell, lftp tracks both your local pwd and your remote pwd, and uses both values (often implicitly) in its commands. By default, your local pwd will be the directory on your local system from where you started the lftp session. You can check your local pwd by prefixing the command with the local keyword

lftp username@server.com:/> local pwd
file:/Volumes/Media/TV/

lftp username@server.com:/> local ls
total 38132360
drwxr-xr-x   4 username  staff   128 Dec  4 14:23 The Big O
-rw-r--r--@  1 username  staff  1170119875 Oct 31  2018 batman.mkv

Upload File

You can upload a single file with the put command. The default behavior is to upload a file from your local pwd to the remote pwd with the same filename

put batman.mkv

This will upload the file batman.mkv from your local pwd to the remote pwd. You can change the filename on the remote side with the -o flag

put batman.mkv -o batman2.mkv
  • if the filename or path has spaces in it, you might need to wrap it in quotes

Upload Directory

The easiest way to upload an entire directory is with the mirror command; mirror sourceRemoteDir targetLocalDir

By default the source is remote and the target is a local directory. When using -R, the source directory is local and the target is remote. If the target directory is omitted, base name of the source directory is used. If both directories are omitted, current local and remote directories are used.

mirror -R "The Big O"

will upload the entire local directory “The Big O” to the remote pwd, with the name “The Big O”

If you instead ran the command

mirror "The Big O"

it would attempt to download the entire remote directory “The Big O” to your local pwd

a command like

mirror remoteDir/ /path/to/localDir/

would download the contents of remoteDir to the local path /path/to/localDir/

It can be a little confusing to remember which direction the mirror will run, so you can preview the transfer without running it by including the --dry-run argument.

Non-Interactive

You can run lftp from the command line non-interactively with the -e and -c flags.

A command like this

lftp -c "pwd; ls" ftp://username:password@server.com

will log into the server, run the pwd and ls commands, then exit.

You can use -e to instead drop you into an interactive session after running the commands. This is really useful for example in situations where you need to pass in some settings in order for your login to work, such as for disabling SSL verification on servers with self-signed certificates

lftp -e "set ssl:verify-certificate false; pwd" ftp://username:password@server.com

This will start a session on the server with SSL verification disabled, print your remote pwd, then enter the interactive session.

Resources

➕
From: hey ✪ | Posted: 2024-01-04 04:42:422024-01-04 04:42 | Filter

Creating a SMB (Samba) Share on Linux (Ubuntu)

Resources:

Creating and configuring a SMB share in Ubuntu is one of the more involved tasks I have done yet. Its not hard, its just a number of steps that need to be completed and are easy to miss or mix up.

The overall process goes like this;

  • create a new user account and user group for sharing (optional but recommended)
  • add your primary user account to the sharing group
  • create the mount point, if it does not already exist, and apply the sharing user group to it recursively
    • also apply group read/write permissions to the directory tree recursively
    • probably a good idea to configure a umask for the dir tree but I keep forgetting to do that
    • IMPORTANT if you are using mergerfs or another disk array file system, you must make sure that your sharing user has rw access to both the array volume and the underlying disks!
  • log in as your sharing user account and verify that you have read and write access to the files and dirs (then log out of the sharing user and back into your primary account)
  • add your sharing user to the Samba SMB daemon
  • create your new SMB mount in the Samba config file
  • give your SMB sharing user access to the SMB share via the Samba daemon config
  • (finally) connect to your SMB share over the network using your sharing user account’s username/password from another Linux system

phew that was a lot… lets see if I can recount all the steps correctly to achieve this; I have them in my bash history but the history is a little out of order after messing up and having to repeat a few steps

Make sure you have Samba installed

sudo apt install samba

Creation of sharing user

# create the "sharing" user group
sudo addgroup sharing

# check that sharing user group we just created actually exists
getent group
# ...
# sharing:x:1002:

# create a user account just for sharing
sudo adduser shareuser
# follow the prompts to create user account password
# you can skip the input for these;
# Full Name []:
# Room Number []:
# Work Phone []:
# Home Phone []:
# Other []:


# log in as the new user to verify
su - shareuser
# as the new shareuser check your groups;
id
# or
groups
# log out with the 'exit' command or Ctrl-D

# as your primary user account

# add the shareuser to the sharing user group
sudo addgroup shareuser sharing

# also add your primary user account to the sharing group for convenience
sudo addgroup username sharing

# verify all the users in the group
sudo getent group sharing

Update Samba SMB Config

# first make a backup copy of the file!
cp /etc/samba/smb.conf smb.conf.bu

# edit the file
sudo nano /etc/samba/smb.conf

Add a section to the file that looks like this

[Media]
path = /mnt/media
valid users = @sharing
browsable = yes
writable = yes
read only = no

where “Media” is the name of your share, and /mnt/media is the path on the host system to the directory to share. The config @sharing gives all users in the “sharing” user group access to the share.

Once you have edited the file, you must test it to make sure there are no errors

testparm

Configure the filesystem for the share

# change the group of the array disks to the new sharing group
sudo chgrp -R sharing /mnt/disk1
sudo chgrp -R sharing /mnt/disk2
sudo chgrp -R sharing /mnt/disk3
sudo chgrp -R sharing /mnt/disk4
sudo chgrp -R sharing /mnt/media # this one is the mergerfs array volume made from the other disks

# change the access permissions to allow for sharing group users to have read and write access
sudo chmod -R 755 /mnt/disk1
sudo chmod -R 755 /mnt/disk2
sudo chmod -R 755 /mnt/disk3
sudo chmod -R 755 /mnt/disk4
sudo chmod -R 755 /mnt/media

# optional: should probably do a umask setting to make sure that these permissions
# propagate automatically to all newly created subdirs of these volumes

# verify that the sharing user has access to the dirs
su - shareuser
# as shareuser;
touch /mnt/disk1/foo
mkdir /mnt/media/bar
touch /mnt/media/bar/foo
# log out of shareuser

# as primary user account

Add sharing user to Samba

The sharing user needs to also be added Samba, and enabled, and supplied with a password.

# add the user
sudo smbpasswd -a shareuser

# enable the user
sudo smbpasswd -e shareuser

# restart the samba daemon
sudo systemctl restart smbd

# check that your shares are active
sudo smbstatus --shares

# verify that your Samba user account is set up
sudo pdbedit -L -v

Connect to your SMB Share from another Linux system

# check that the SMB share is available
# insert the IP address to the other server here;
smbclient -L 192.168.1.2 -U shareuser

# test out the mount manually
# make sure that /mnt/media exists on the local system already with 'sudo mkdir -p' and that you have ownership of the dir first!! Before you run this
sudo mount -t cifs -o username=shareuser //192.168.1.2/Media /mnt/media

# if it worked then unmount it;
sudo umount /mnt/media

# edit your fstab
# good idea to make a backup copy first!
sudo nano /etc/fstab

Add an entry like this to your fstab

//192.168.1.2/Media /mnt/media cifs username=shareuser,password=putpasswordhere,uid=username,gid=username,vers=3.0 0 0
  • you can embed the username and password for the shareuser here or you can save it to an external credential file
  • for convenience you can set the uid to your primary user account and the gid to your primary user account’s user group; this will apply those permissions to the mounted volume, making it easier to interact with the volume (read and write) from within your Docker container which are running under your primary user account

Try to mount the new fstab

sudo mount -a

# if you get errors, check
sudo dmesg

Finally, you should verify that it works;

touch /mnt/media/foo

Notes

If you start having permissions issues, you need to verify that the permissions are set correctly for all dirs and subdirs and files

  • from the SMB host “shareuser” on the individual mergerfs array disks
  • on the mergerfs array volume
  • on the SMB client’s mounted volume

Make sure to propagate the SMB client UID and GID into your Docker containers correctly in the Docker compose file.

Make sure to mount the volume into your Docker containers with /mnt/media:/mnt/media; preserving the exact same file path from your SMB host to your SMB client to inside your Docker containers will save you a lot of path-mapping headaches.

To verify that your docker containers have correct permissions, you can enter them and test it out;

docker exec -ti radarr bash
# inside the docker container

# check that you can see the mounted volume
ls -l /mnt/

# check that you can write to the mounted volume
touch /mnt/media/foo

# potentially repeat ^^^ for other subdirs as needed
➕
From: hey ✪ | Posted: 2024-01-04 15:02:212024-01-04 15:02 | Filter

ok5dmzrtdfac1.jpeg

➕
From: hey ✪ | Posted: 2024-01-16 19:17:592024-01-16 19:17 | Filter
➕
From: hey ✪ | Posted: 2024-02-13 19:23:292024-02-13 19:23 | Filter
➕
Reply:
Autoscroll
Forget
To reply to this thread, please join this community.
1 person is reading this thread now.
← Thread List | ↑ Top  𝐀   ⬆︎