Merge pull request #117 from Gabisonfire/tidy-optional-extras

Attempt to tidy up Caddy and Metrics

.gitignore

@@ -407,3 +407,7 @@ dist/
 deployment/docker/docker-compose-dev.yaml
 
 src/producer/.run/
+
+# Caddy logs
+deployment/docker/optional_reverse_proxy/logs/**
+!deployment/docker/optional_reverse_proxy/logs/.gitkeep

README.md

@@ -12,27 +12,29 @@
 
 > [!IMPORTANT]
 > The latest change renames the project and requires a [small migration](#selfhostio-to-knightcrawler-migration).
-- [Knight Crawler](#knight-crawler)
-  - [Contents](#contents)
-  - [Overview](#overview)
-  - [Using](#using)
-    - [Download Docker and Docker Compose v2](#download-docker-and-docker-compose-v2)
-    - [Environment Setup](#environment-setup)
-      - [Optional Configuration Changes](#optional-configuration-changes)
-    - [DebridMediaManager setup (optional)](#debridmediamanager-setup-optional)
-    - [Configure external access](#configure-external-access)
-    - [Run the project](#run-the-project)
-    - [Monitoring with Grafana and Prometheus (Optional)](#monitoring-with-grafana-and-prometheus-optional)
-      - [Accessing RabbitMQ Management](#accessing-rabbitmq-management)
-      - [Using Grafana and Prometheus](#using-grafana-and-prometheus)
-  - [Importing external dumps](#importing-external-dumps)
-    - [Importing data into PostgreSQL](#importing-data-into-postgresql)
-      - [Using pgloader via docker](#using-pgloader-via-docker)
-      - [Using native installation of pgloader](#using-native-installation-of-pgloader)
-    - [Process the data we have imported](#process-the-data-we-have-imported)
-    - [I imported the data without the `LIKE 'movies%%'` queries!](#i-imported-the-data-without-the-like-movies-queries)
-  - [Selfhostio to KnightCrawler Migration](#selfhostio-to-knightcrawler-migration)
-  - [To-do](#to-do)
+- [Contents](#contents)
+- [Overview](#overview)
+- [Using](#using)
+  - [Download Docker and Docker Compose v2](#download-docker-and-docker-compose-v2)
+  - [Environment Setup](#environment-setup)
+    - [Optional Configuration Changes](#optional-configuration-changes)
+  - [DebridMediaManager setup (optional)](#debridmediamanager-setup-optional)
+  - [Configure external access](#configure-external-access)
+    - [I have a public IP address and can open ports](#i-have-a-public-ip-address-and-can-open-ports)
+    - [I will be using a tunnel/vpn (CGNAT, don't want to open ports, etc...)](#i-will-be-using-a-tunnelvpn-cgnat-dont-want-to-open-ports-etc)
+    - [Next steps](#next-steps)
+  - [Run the project](#run-the-project)
+  - [Monitoring with Grafana and Prometheus (Optional)](#monitoring-with-grafana-and-prometheus-optional)
+    - [Accessing RabbitMQ Management](#accessing-rabbitmq-management)
+    - [Using Grafana and Prometheus](#using-grafana-and-prometheus)
+- [Importing external dumps](#importing-external-dumps)
+  - [Importing data into PostgreSQL](#importing-data-into-postgresql)
+    - [Using pgloader via docker](#using-pgloader-via-docker)
+    - [Using native installation of pgloader](#using-native-installation-of-pgloader)
+  - [Process the data we have imported](#process-the-data-we-have-imported)
+  - [I imported the data without the `LIKE 'movies%%'` queries!](#i-imported-the-data-without-the-like-movies-queries)
+- [Selfhostio to KnightCrawler Migration](#selfhostio-to-knightcrawler-migration)
+- [To-do](#to-do)
 
 
 ## Overview
@@ -97,30 +99,87 @@ We can search DebridMediaManager hash lists which are hosted on GitHub. This all
    ```
 ### Configure external access
 
-What you will need:
-1. Domain or subdomain that points toward your IP. You can use [DuckDNS](duckdns.org) for a free subdomain. [Installation instructions](http://www.duckdns.org/install.jsp) are provided to keep your IP updated.
-2. Ports 80 and 443 opened on your router/gateway and forwarded to your Knightcrawler server. Refer to [PortForward.com](https://portforward.com/). Please note that this action may pose security vulnerabilities and potential damage for which Knightcrawler and its contributors cannot be held responsible.
+Please choose which applies to you:
 
-Navigate to `knightcrawler/development/docker` and edit the `Caddyfile` to replace `your-domain.com` with your domain name.
-
-### Run the project
+- [I have a public IP address and can open ports](#i-have-a-public-ip-address-and-can-open-ports)
+- [I will be using a tunnel/vpn (CGNAT, don't want to open ports, etc...)](#i-will-be-using-a-tunnelvpn-cgnat-dont-want-to-open-ports-etc)
 
 
- If you have configured external access, utilize the following commands:
-```sh
-cd deployment/docker
-docker compose -f docker-compose.yaml -f docker-compose-caddy.yaml up -d
+#### I have a public IP address and can open ports
+
+You can use either a paid domain `your-domain.com` or a free reverse dns service like [DuckDNS](https://www.duckdns.org/) (you can [automate the update of your IP address](https://www.duckdns.org/install.jsp)).
+
+Before continuing you need to open up port `80` and `443` in your firewall and configure any [port forwarding](https://portforward.com/) as necessary. You should not do this unless you understand the security implications. Please note that Knightcrawler and its contributors cannot be held responsible for any damage or loss of data from exposing your service publically.
+
+You may find it safer to [use a tunnel/vpn](#i-will-be-using-a-tunnelvpn-cgnat-dont-want-to-open-ports-etc), but this will require the use of a paid domain or will not be accessible without being connected to your vpn.
+
+#### I will be using a tunnel/vpn (CGNAT, don't want to open ports, etc...)
+
+For this you can use a VPN like [Tailscale](https://tailscale.com/) which has its own ways of issuing SSL certs, or you can use a tunnel like [Cloudflare](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/).
+
+To use a Cloudflare tunnel you __will__ need a domain name.
+
+Theres a sample compose for a Cloudflare tunnel [here](deployment/docker/example_cloudflare_tunnel/docker-compose.yml).
+
+If you are going to go this route, you will want to connect caddy to the cloudflare-tunnel network. It's all in Caddy's [docker-compose.yaml](deployment/docker/optional_reverse_proxy/docker-compose.yaml) you will just need to uncomment it.
+
+#### Next steps
+
+Regardless of what method you choose, you will need to connect Knight Crawler to Caddy. We only need to expose the addon, the rest of the services can remain internal.
+
+In our primary [docker-compose.yaml](deployment/docker/docker-compose.yaml) we will add the Caddy network:
+
+```
+networks:
+  knightcrawler-network:
+    driver: bridge
+    name: knightcrawler-network
+
+  caddy:
+    name: caddy
+    external: true
 ```
 
- If you have not configured external access, utilize the following commands:
+Remove or comment out the port for the addon, and connect it to Caddy:
+
+```
+addon:
+  <<: *knightcrawler-app
+  env_file:
+    - .env
+  hostname: knightcrawler-addon
+  image: gabisonfire/knightcrawler-addon:latest
+  labels:
+    logging: "promtail"
+  networks:
+    - knightcrawler-network
+    - caddy         # <~~~~~~~ These lines
+  # ports:          # <~~~~~~~ have been
+  #   - "7000:7000" # <~~~~~~~ changed
+```
+
+If you are using a Cloudflare tunnel, start it before Caddy.
+
+Caddy can be started with:
+
+```sh
+cd deployment/docker/optional_reverse_proxy
+docker compose up -d
+```
+
+It should be started before Knight Crawler.
+
+### Run the project
+To start the project use the following commands:
+
 ```sh
 cd deployment/docker
 docker compose up -d
 ```
 
-It will take a while to find and add the torrents to the database. During initial testing, in one hour it's estimated that around 200,000 torrents were located and added to the queue to be processed. For best results, you should leave everything running for a few hours.
+It will take a while to find and add the torrents to the database. During initial testing, in one hour it's estimated that around 200,000 torrents were located and added to the queue to be processed. The processing takes longer, unfortunately and you may not find the movie/show you want for a while. For best results, you should leave everything running for a few hours.
 
-To add the addon to Stremio, open a web browser and navigate to: [http://127.0.0.1:7000](http://127.0.0.1:7000)
+To add the addon to Stremio, open a web browser and navigate to: [http://127.0.0.1:7000](http://127.0.0.1:7000) or [knightcrawler.your-domain.com](https://knightcrawler.your-domain.com) if you are using Caddy.
 
 ### Monitoring with Grafana and Prometheus (Optional)
 
@@ -128,7 +187,6 @@ To enhance your monitoring capabilities, you can use Grafana and Prometheus in a
 
 #### Accessing RabbitMQ Management
 
-
 You can still monitor RabbitMQ by accessing its management interface at [http://127.0.0.1:15672/](http://127.0.0.1:15672/). Use the provided credentials to log in and explore RabbitMQ's monitoring features (the default username and password are `guest`).
 
 #### Using Grafana and Prometheus
@@ -138,8 +196,8 @@ Here's how to set up and use Grafana and Prometheus for monitoring RabbitMQ:
 1. **Start Grafana and Prometheus**: Run the following command to start both Grafana and Prometheus:
 
    ```sh
-   cd deployment/docker
-   docker compose -f docker-compose-metrics.yml up -d
+   cd deployment/docker/optional_metrics
+   docker compose up -d
    ```
 
    - Grafana will be available at [http://127.0.0.1:3000](http://127.0.0.1:3000).

deployment/docker/Caddyfile

@@ -1,3 +0,0 @@
-your-domain.com {
-	reverse_proxy 127.0.0.1:7000
-}

deployment/docker/docker-compose-caddy.yaml

@@ -1,21 +0,0 @@
-version: "3.7"
-
-services:
-  caddy:
-    image: caddy:latest
-    restart: unless-stopped
-    cap_add:
-      - NET_ADMIN
-    ports:
-      - "80:80"
-      - "443:443"
-      - "443:443/udp"
-    volumes:
-      - $PWD/Caddyfile:/etc/caddy/Caddyfile
-      - $PWD/site:/srv
-      - caddy_data:/data
-      - caddy_config:/config
-
-volumes:
-  caddy_data:
-  caddy_config:

deployment/docker/docker-compose.yaml

@@ -116,23 +116,29 @@ services:
     networks:
       - knightcrawler-network
 
-  addon:
-    image: gabisonfire/knightcrawler-addon:latest
-    ports:
-      - "7000:7000"
-    labels:
-      logging: "promtail"
-    env_file:
-      - .env
-    <<: *knightcrawler-app
-    networks:
-      - knightcrawler-network
+addon:
+  <<: *knightcrawler-app
+  env_file:
+    - .env
+  hostname: knightcrawler-addon
+  image: gabisonfire/knightcrawler-addon:latest
+  labels:
+    logging: "promtail"
+  networks:
+    - knightcrawler-network
+  ports:
+    - "7000:7000"
+
       
 networks:
   knightcrawler-network:
     driver: bridge
     name: knightcrawler-network
 
+  caddy:
+    name: caddy
+    external: true
+
 volumes:
   postgres:
   mongo:

deployment/docker/example_cloudflare_tunnel/.env.example

@@ -0,0 +1 @@
+TOKEN=cloudflare-tunnel-token-here

deployment/docker/example_cloudflare_tunnel/docker-compose.yml

@@ -0,0 +1,18 @@
+name: cloudflared
+
+networks:
+  cloudflare-tunnel:
+    name: cloudflare-tunnel
+    external: true
+
+services:
+  cloudflared:
+    container_name: cloudflared
+    command: tunnel --no-autoupdate run --token ${TOKEN}
+    hostname: cloudflared
+    image: cloudflare/cloudflared:latest
+    networks:
+      - cloudflare-tunnel
+    restart: always
+    security_opt:
+      - no-new-privileges:true

deployment/docker/optional_metrics/docker-compose.yml

@@ -1,74 +1,76 @@
-version: '3.8'
 name: knightcrawler-metrics
 
+networks:
+  knightcrawler-network:
+    external: true
+
+volumes:
+  grafana-data:
+  loki-data:
+    
 services:
   prometheus:
-    image: prom/prometheus:v2.20.1
-    volumes:
-      - ./config/prometheus/config.yml:/etc/prometheus/prometheus.yml
     command:
       - '--config.file=/etc/prometheus/prometheus.yml'
-    ports:
-      - "9090:9090"
+    image: prom/prometheus:v2.20.1
     networks:
       - knightcrawler-network
-    
-  grafana:
-    image: grafana/grafana:latest
+    ports:
+      - 9090:9090
     volumes:
-      - ./config/grafana/datasources:/etc/grafana/provisioning/datasources
-      - ./config/grafana/dashboards/dashboards.yml:/etc/grafana/provisioning/dashboards/dashboards.yml
-      - ./config/grafana/dashboards/logs.json:/var/lib/grafana/dashboards/logs.json
-      - grafana-data:/var/lib/grafana
-    ports:
-      - "3000:3000"
-    environment:
-      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
-      - GF_SECURITY_ADMIN_USER=admin
-      - GF_SECURITY_ADMIN_PASSWORD=admin_password
-    depends_on:
-      - prometheus
-    networks:
-      - knightcrawler-network
+      - ./config/prometheus/config.yml:/etc/prometheus/prometheus.yml
+
+grafana:
+  depends_on:
+    - prometheus
+  environment:
+    - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
+    - GF_SECURITY_ADMIN_PASSWORD=admin_password
+    - GF_SECURITY_ADMIN_USER=admin
+  image: grafana/grafana:latest
+  networks:
+    - knightcrawler-network
+  ports:
+    - "3000:3000"
+  volumes:
+    - ./config/grafana/dashboards/dashboards.yml:/etc/grafana/provisioning/dashboards/dashboards.yml
+    - ./config/grafana/dashboards/logs.json:/var/lib/grafana/dashboards/logs.json
+    - ./config/grafana/datasources:/etc/grafana/provisioning/datasources
+    - grafana-data:/var/lib/grafana
 
   postgres-exporter:
-    image: prometheuscommunity/postgres-exporter
     env_file:
-      - .env
+      - ../.env
     environment:
       DATA_SOURCE_NAME: "postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}?sslmode=disable"
+    image: prometheuscommunity/postgres-exporter
     networks:
       - knightcrawler-network
       
   promtail:
     image: grafana/promtail:2.9.4
     volumes:
+      - ./config/promtail/config.yml:/etc/promtail/config.yml
       - /var/lib/docker/containers:/var/lib/docker/containers:ro
       - /var/run/docker.sock:/var/run/docker.sock
-      - ./config/promtail/config.yml:/etc/promtail/config.yml
-    command: -config.file=/etc/promtail/config.yml
+    command:
+      - '-config.file=/etc/promtail/config.yml'
     depends_on:
       - prometheus
       - loki
     networks:
       - knightcrawler-network
+
       
   loki:
-    image: grafana/loki:2.9.4
-    command: -config.file=/etc/loki/local-config.yml
+    command: '-config.file=/etc/loki/local-config.yml'
     depends_on:
-      - prometheus
       - grafana
-    volumes:
-      - loki-data:/loki
-      - ./config/loki/config.yml:/etc/loki/local-config.yml
+      - prometheus
+    image: grafana/loki:2.9.4
     networks:
       - knightcrawler-network
-        
-volumes:
-  loki-data:
-  grafana-data:
-      
-networks:
-  knightcrawler-network:
-    external: true
+    volumes:
+      - ./config/loki/config.yml:/etc/loki/local-config.yml
+      - loki-data:/loki
+

deployment/docker/optional_reverse_proxy/configs/Caddyfile

@@ -0,0 +1,62 @@
+{
+	## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+	## Let's Encrpyt staging environment
+	## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+	## Once you have confirmed Caddy works you should comment out
+	## the below line:
+	acme_ca https://acme-staging-v02.api.letsencrypt.org/director
+}
+
+(security-headers) {
+	## These are the recommended default settings in Caddy documentation
+	## https://caddyserver.com/docs/caddyfile/directives/header
+	header {
+		## disable FLoC tracking
+		Permissions-Policy "interest-cohort=()"
+
+		## enable HSTS
+		Strict-Transport-Security "max-age=300;" # 5 minutes
+		## NOTE: I have dramatically lowered the above for testing.
+		## Once you have confirmed that everything works, start increasing the number
+		## the goal is to have HSTS set to one year with subdomains and preloading :
+		## 
+		# `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload`
+		## 
+		## Warning: You should ensure that you fully understand the implications
+		## of HSTS preloading before you include the directive in your policy and
+		## before you submit. It means that your entire domain and all subdomains,
+		## including those managed or maintained by third parties, will only work
+		## with HTTPS. Preloading should be viewed as a one way ticket.
+		## Whilst it is possible to be removed, it can take a long time and you
+		## may not be removed from all browsers.
+
+		## disable clients from sniffing the media type
+		X-Content-Type-Options "nosniff"
+
+		## clickjacking protection
+		X-Frame-Options "DENY"
+	}
+}
+
+(cloudflare-tunnel-protection) {
+	import ./snippets/cloudflare-replace-X-Forwarded-For
+	trusted_proxies 172.17.0.0/16 # This needs to be your docker subnet
+	# I beleive this is what is configured by default.
+	# If you can't make it work ask for my help on discord.
+}
+
+knightcrawler.your-domain.com {
+	## Uncomment to enable logging
+	# log {
+	# 	output file /var/log/caddy/knightcrawler.your-domain.com.log {
+	# 		roll_size 10mb
+	# 		roll_keep 5
+	# 		roll_keep_for 720h
+	# 	}
+	# }
+
+	encode gzip
+	## DO NOT ENABLE UNTIL YOU HAVE DISABLED THE TESTING ENVIRONMENT
+	# import security-headers
+	reverse_proxy knightcrawler-addon:7000
+}

deployment/docker/optional_reverse_proxy/configs/snippets/cloudflare-replace-X-Forwarded-For

@@ -0,0 +1,2 @@
+header_up X-Real-IP {http.request.header.CF-Connecting-IP}
+header_up X-Forwarded-For {http.request.header.CF-Connecting-IP}

deployment/docker/optional_reverse_proxy/docker-compose.yaml

@@ -0,0 +1,39 @@
+name: caddy
+
+networks:
+  caddy:
+    name: caddy
+    external: true
+
+  # cloudflare-tunnel:
+  #   name: cloudflare-tunnel
+  #   external: true
+
+volumes:
+  config:
+  data:
+
+services:
+  # Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS
+  # https://github.com/caddyserver/caddy
+  caddy:
+    cap_add:
+      - NET_ADMIN
+    container_name: caddy
+    hostname: caddy
+    image: caddy:2.7.6 # May be out of date, but using the `:latest` tag is how you accidentally break your server
+    networks:
+      - caddy
+      # - cloudflare-tunnel
+    ports:
+      - 80:80
+      - 443:443
+      - 443:443/udp
+    restart: always
+    security_opt:
+      - no-new-privileges:true
+    volumes:
+      - ./configs/:/etc/caddy/ # /etc/caddy/Caddyfile and /etc/caddy/snippets/
+      - ./logs:/var/log/caddy/
+      - config:/config
+      - data:/data