brock/torrentio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Attempt to tidy up Caddy and Metrics

Browse Source
This commit is contained in:
purple_emily
2024-03-05 10:20:25 +00:00
parent 7d5fbb2871
commit 0b12a8cbc2
19 changed files with 285 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
.gitignore vendored
View File

@@ -407,3 +407,11 @@ dist/
deployment/docker/docker-compose-dev.yaml
src/producer/.run/
# Git being annoying.
# Don't ignore the logs folder.
# Ignore everything inside it.
# Don't ignore the .gitkeep file
!deployment/docker/optional_reverse_proxy/logs/
deployment/docker/optional_reverse_proxy/logs/*
!deployment/docker/optional_reverse_proxy/logs/.gitkeep

132
README.md
View File

@@ -12,27 +12,29 @@
> [!IMPORTANT]
> The latest change renames the project and requires a [small migration](#selfhostio-to-knightcrawler-migration).
- [Knight Crawler](#knight-crawler)
- [Contents](#contents)
- [Overview](#overview)
- [Using](#using)
- [Download Docker and Docker Compose v2](#download-docker-and-docker-compose-v2)
- [Environment Setup](#environment-setup)
- [Optional Configuration Changes](#optional-configuration-changes)
- [DebridMediaManager setup (optional)](#debridmediamanager-setup-optional)
- [Configure external access](#configure-external-access)
- [Run the project](#run-the-project)
- [Monitoring with Grafana and Prometheus (Optional)](#monitoring-with-grafana-and-prometheus-optional)
- [Accessing RabbitMQ Management](#accessing-rabbitmq-management)
- [Using Grafana and Prometheus](#using-grafana-and-prometheus)
- [Importing external dumps](#importing-external-dumps)
- [Importing data into PostgreSQL](#importing-data-into-postgresql)
- [Using pgloader via docker](#using-pgloader-via-docker)
- [Using native installation of pgloader](#using-native-installation-of-pgloader)
- [Process the data we have imported](#process-the-data-we-have-imported)
- [I imported the data without the `LIKE 'movies%%'` queries!](#i-imported-the-data-without-the-like-movies-queries)
- [Selfhostio to KnightCrawler Migration](#selfhostio-to-knightcrawler-migration)
- [To-do](#to-do)
- [Contents](#contents)
- [Overview](#overview)
- [Using](#using)
- [Download Docker and Docker Compose v2](#download-docker-and-docker-compose-v2)
- [Environment Setup](#environment-setup)
- [Optional Configuration Changes](#optional-configuration-changes)
- [DebridMediaManager setup (optional)](#debridmediamanager-setup-optional)
- [Configure external access](#configure-external-access)
- [I have a public IP address and can open ports](#i-have-a-public-ip-address-and-can-open-ports)
- [I will be using a tunnel/vpn (CGNAT, don't want to open ports, etc...)](#i-will-be-using-a-tunnelvpn-cgnat-dont-want-to-open-ports-etc)
- [Next steps](#next-steps)
- [Run the project](#run-the-project)
- [Monitoring with Grafana and Prometheus (Optional)](#monitoring-with-grafana-and-prometheus-optional)
- [Accessing RabbitMQ Management](#accessing-rabbitmq-management)
- [Using Grafana and Prometheus](#using-grafana-and-prometheus)
- [Importing external dumps](#importing-external-dumps)
- [Importing data into PostgreSQL](#importing-data-into-postgresql)
- [Using pgloader via docker](#using-pgloader-via-docker)
- [Using native installation of pgloader](#using-native-installation-of-pgloader)
- [Process the data we have imported](#process-the-data-we-have-imported)
- [I imported the data without the `LIKE 'movies%%'` queries!](#i-imported-the-data-without-the-like-movies-queries)
- [Selfhostio to KnightCrawler Migration](#selfhostio-to-knightcrawler-migration)
- [To-do](#to-do)
## Overview
@@ -97,30 +99,87 @@ We can search DebridMediaManager hash lists which are hosted on GitHub. This all
```
### Configure external access
What you will need:
1. Domain or subdomain that points toward your IP. You can use [DuckDNS](duckdns.org) for a free subdomain. [Installation instructions](http://www.duckdns.org/install.jsp) are provided to keep your IP updated.
2. Ports 80 and 443 opened on your router/gateway and forwarded to your Knightcrawler server. Refer to [PortForward.com](https://portforward.com/). Please note that this action may pose security vulnerabilities and potential damage for which Knightcrawler and its contributors cannot be held responsible.
Please choose which applies to you:
Navigate to `knightcrawler/development/docker` and edit the `Caddyfile` to replace `your-domain.com` with your domain name.
### Run the project
- [I have a public IP address and can open ports](#i-have-a-public-ip-address-and-can-open-ports)
- [I will be using a tunnel/vpn (CGNAT, don't want to open ports, etc...)](#i-will-be-using-a-tunnelvpn-cgnat-dont-want-to-open-ports-etc)
If you have configured external access, utilize the following commands:
```sh
cd deployment/docker
docker compose -f docker-compose.yaml -f docker-compose-caddy.yaml up -d
### I have a public IP address and can open ports
You can use either a paid domain `your-domain.com` or a free reverse dns service like [DuckDNS](https://www.duckdns.org/) (you can [automate the update of your IP address](https://www.duckdns.org/install.jsp)).
Before continuing you need to open up port `80` and `443` in your firewall and configure any [port forwarding](https://portforward.com/) as necessary. You should not do this unless you understand the security implications. Please note that Knightcrawler and its contributors cannot be held responsible for any damage or loss of data from exposing your service publically.
You may find it safer to [use a tunnel/vpn](#i-will-be-using-a-tunnelvpn-cgnat-dont-want-to-open-ports-etc), but this will require the use of a paid domain or will not be accessible without being connected to your vpn.
### I will be using a tunnel/vpn (CGNAT, don't want to open ports, etc...)
For this you can use a VPN like [Tailscale](https://tailscale.com/) which has its own ways of issuing SSL certs, or you can use a tunnel like [Cloudflare](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/).
To use a Cloudflare tunnel you __will__ need a domain name.
Theres a sample compose for a Cloudflare tunnel [here](deployment/docker/example_cloudflare_tunnel/docker-compose.yml).
If you are going to go this route, you will want to connect caddy to the cloudflare-tunnel network. It's all in Caddy's [docker-compose.yaml](deployment/docker/optional_reverse_proxy/docker-compose.yaml) you will just need to uncomment it.
### Next steps
Regardless of what method you choose, you will need to connect Knight Crawler to Caddy. We only need to expose the addon, the rest of the services can remain internal.
In our primary [docker-compose.yaml](deployment/docker/docker-compose.yaml) we will add the Caddy network:
```
networks:
knightcrawler-network:
driver: bridge
name: knightcrawler-network
caddy:
name: caddy
external: true
```
If you have not configured external access, utilize the following commands:
Remove or comment out the port for the addon, and connect it to Caddy:
```
addon:
<<: *knightcrawler-app
env_file:
- .env
hostname: knightcrawler-addon
image: gabisonfire/knightcrawler-addon:latest
labels:
logging: "promtail"
networks:
- knightcrawler-network
- caddy # <~~~~~~~ These lines
# ports: # <~~~~~~~ have been
# - "7000:7000" # <~~~~~~~ changed
```
If you are using a Cloudflare tunnel, start it before Caddy.
Caddy can be started with:
```sh
cd deployment/docker/optional_reverse_proxy
docker compose up -d
```
It should be started before Knight Crawler.
### Run the project
To start the project use the following commands:
```sh
cd deployment/docker
docker compose up -d
```
It will take a while to find and add the torrents to the database. During initial testing, in one hour it's estimated that around 200,000 torrents were located and added to the queue to be processed. For best results, you should leave everything running for a few hours.
It will take a while to find and add the torrents to the database. During initial testing, in one hour it's estimated that around 200,000 torrents were located and added to the queue to be processed. The processing takes longer, unfortunately and you may not find the movie/show you want for a while. For best results, you should leave everything running for a few hours.
To add the addon to Stremio, open a web browser and navigate to: [http://127.0.0.1:7000](http://127.0.0.1:7000)
To add the addon to Stremio, open a web browser and navigate to: [http://127.0.0.1:7000](http://127.0.0.1:7000) or [knightcrawler.your-domain.com](https://knightcrawler.your-domain.com) if you are using Caddy.
### Monitoring with Grafana and Prometheus (Optional)
@@ -128,7 +187,6 @@ To enhance your monitoring capabilities, you can use Grafana and Prometheus in a
#### Accessing RabbitMQ Management
You can still monitor RabbitMQ by accessing its management interface at [http://127.0.0.1:15672/](http://127.0.0.1:15672/). Use the provided credentials to log in and explore RabbitMQ's monitoring features (the default username and password are `guest`).
#### Using Grafana and Prometheus
@@ -138,8 +196,8 @@ Here's how to set up and use Grafana and Prometheus for monitoring RabbitMQ:
1. **Start Grafana and Prometheus**: Run the following command to start both Grafana and Prometheus:
```sh
cd deployment/docker
docker compose -f docker-compose-metrics.yml up -d
cd deployment/docker/optional_metrics
docker compose up -d
```
- Grafana will be available at [http://127.0.0.1:3000](http://127.0.0.1:3000).

3
deployment/docker/Caddyfile
View File

@@ -1,3 +0,0 @@
your-domain.com {
reverse_proxy 127.0.0.1:7000
}

21
deployment/docker/docker-compose-caddy.yaml
View File

@@ -1,21 +0,0 @@
version: "3.7"
services:
caddy:
image: caddy:latest
restart: unless-stopped
cap_add:
- NET_ADMIN
ports:
- "80:80"
- "443:443"
- "443:443/udp"
volumes:
- $PWD/Caddyfile:/etc/caddy/Caddyfile
- $PWD/site:/srv
- caddy_data:/data
- caddy_config:/config
volumes:
caddy_data:
caddy_config:

28
deployment/docker/docker-compose.yaml
View File

@@ -116,23 +116,29 @@ services:
networks:
- knightcrawler-network
addon:
image: gabisonfire/knightcrawler-addon:latest
ports:
- "7000:7000"
labels:
logging: "promtail"
env_file:
- .env
<<: *knightcrawler-app
networks:
- knightcrawler-network
addon:
<<: *knightcrawler-app
env_file:
- .env
hostname: knightcrawler-addon
image: gabisonfire/knightcrawler-addon:latest
labels:
logging: "promtail"
networks:
- knightcrawler-network
ports:
- "7000:7000"
networks:
knightcrawler-network:
driver: bridge
name: knightcrawler-network
caddy:
name: caddy
external: true
volumes:
postgres:
mongo:

1
deployment/docker/example_cloudflare_tunnel/.env.example Normal file
View File

@@ -0,0 +1 @@
TOKEN=cloudflare-tunnel-token-here

18
deployment/docker/example_cloudflare_tunnel/docker-compose.yml Normal file
View File

@@ -0,0 +1,18 @@
name: cloudflared
networks:
cloudflare-tunnel:
name: cloudflare-tunnel
external: true
services:
cloudflared:
container_name: cloudflared
command: tunnel --no-autoupdate run --token ${TOKEN}
hostname: cloudflared
image: cloudflare/cloudflared:latest
networks:
- cloudflare-tunnel
restart: always
security_opt:
- no-new-privileges:true

0
deployment/docker/config/grafana/dashboards/dashboards.yml → deployment/docker/optional_metrics/config/grafana/dashboards/dashboards.yml
View File

0
deployment/docker/config/grafana/dashboards/logs.json → deployment/docker/optional_metrics/config/grafana/dashboards/logs.json
View File

0
deployment/docker/config/grafana/datasources/loki-datasource.yml → deployment/docker/optional_metrics/config/grafana/datasources/loki-datasource.yml
View File

0
deployment/docker/config/grafana/datasources/prometheus-datasource.yml → deployment/docker/optional_metrics/config/grafana/datasources/prometheus-datasource.yml
View File

0
deployment/docker/config/loki/config.yml → deployment/docker/optional_metrics/config/loki/config.yml
View File

0
deployment/docker/config/prometheus/config.yml → deployment/docker/optional_metrics/config/prometheus/config.yml
View File

0
deployment/docker/config/promtail/config.yml → deployment/docker/optional_metrics/config/promtail/config.yml
View File

84
deployment/docker/docker-compose-metrics.yml → deployment/docker/optional_metrics/docker-compose.yml
View File

@@ -1,74 +1,76 @@
version: '3.8'
name: knightcrawler-metrics
networks:
knightcrawler-network:
external: true
volumes:
grafana-data:
loki-data:
services:
prometheus:
image: prom/prometheus:v2.20.1
volumes:
- ./config/prometheus/config.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
ports:
- "9090:9090"
image: prom/prometheus:v2.20.1
networks:
- knightcrawler-network
grafana:
image: grafana/grafana:latest
ports:
- 9090:9090
volumes:
- ./config/grafana/datasources:/etc/grafana/provisioning/datasources
- ./config/grafana/dashboards/dashboards.yml:/etc/grafana/provisioning/dashboards/dashboards.yml
- ./config/grafana/dashboards/logs.json:/var/lib/grafana/dashboards/logs.json
- grafana-data:/var/lib/grafana
ports:
- "3000:3000"
environment:
- GF_PATHS_PROVISIONING=/etc/grafana/provisioning
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin_password
depends_on:
- prometheus
networks:
- knightcrawler-network
- ./config/prometheus/config.yml:/etc/prometheus/prometheus.yml
grafana:
depends_on:
- prometheus
environment:
- GF_PATHS_PROVISIONING=/etc/grafana/provisioning
- GF_SECURITY_ADMIN_PASSWORD=admin_password
- GF_SECURITY_ADMIN_USER=admin
image: grafana/grafana:latest
networks:
- knightcrawler-network
ports:
- "3000:3000"
volumes:
- ./config/grafana/dashboards/dashboards.yml:/etc/grafana/provisioning/dashboards/dashboards.yml
- ./config/grafana/dashboards/logs.json:/var/lib/grafana/dashboards/logs.json
- ./config/grafana/datasources:/etc/grafana/provisioning/datasources
- grafana-data:/var/lib/grafana
postgres-exporter:
image: prometheuscommunity/postgres-exporter
env_file:
- .env
- ../.env
environment:
DATA_SOURCE_NAME: "postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}?sslmode=disable"
image: prometheuscommunity/postgres-exporter
networks:
- knightcrawler-network
promtail:
image: grafana/promtail:2.9.4
volumes:
- ./config/promtail/config.yml:/etc/promtail/config.yml
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock
- ./config/promtail/config.yml:/etc/promtail/config.yml
command: -config.file=/etc/promtail/config.yml
command:
- '-config.file=/etc/promtail/config.yml'
depends_on:
- prometheus
- loki
networks:
- knightcrawler-network
loki:
image: grafana/loki:2.9.4
command: -config.file=/etc/loki/local-config.yml
command: '-config.file=/etc/loki/local-config.yml'
depends_on:
- prometheus
- grafana
volumes:
- loki-data:/loki
- ./config/loki/config.yml:/etc/loki/local-config.yml
- prometheus
image: grafana/loki:2.9.4
networks:
- knightcrawler-network
volumes:
loki-data:
grafana-data:
networks:
knightcrawler-network:
external: true
volumes:
- ./config/loki/config.yml:/etc/loki/local-config.yml
- loki-data:/loki

62
deployment/docker/optional_reverse_proxy/configs/Caddyfile Normal file
View File

@@ -0,0 +1,62 @@
{
## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
## Let's Encrpyt staging environment
## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
## Once you have confirmed Caddy works you should comment out
## the below line:
acme_ca https://acme-staging-v02.api.letsencrypt.org/director
}
(security-headers) {
## These are the recommended default settings in Caddy documentation
## https://caddyserver.com/docs/caddyfile/directives/header
header {
## disable FLoC tracking
Permissions-Policy "interest-cohort=()"
## enable HSTS
Strict-Transport-Security "max-age=300;" # 5 minutes
## NOTE: I have dramatically lowered the above for testing.
## Once you have confirmed that everything works, start increasing the number
## the goal is to have HSTS set to one year with subdomains and preloading :
##
# `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload`
##
## Warning: You should ensure that you fully understand the implications
## of HSTS preloading before you include the directive in your policy and
## before you submit. It means that your entire domain and all subdomains,
## including those managed or maintained by third parties, will only work
## with HTTPS. Preloading should be viewed as a one way ticket.
## Whilst it is possible to be removed, it can take a long time and you
## may not be removed from all browsers.
## disable clients from sniffing the media type
X-Content-Type-Options "nosniff"
## clickjacking protection
X-Frame-Options "DENY"
}
}
(cloudflare-tunnel-protection) {
import ./snippets/cloudflare-replace-X-Forwarded-For
trusted_proxies 172.17.0.0/16 # This needs to be your docker subnet
# I beleive this is what is configured by default.
# If you can't make it work ask for my help on discord.
}
knightcrawler.your-domain.com {
## Uncomment to enable logging
# log {
# output file /var/log/caddy/knightcrawler.your-domain.com.log {
# roll_size 10mb
# roll_keep 5
# roll_keep_for 720h
# }
# }
encode gzip
## DO NOT ENABLE UNTIL YOU HAVE DISABLED THE TESTING ENVIRONMENT
# import security-headers
reverse_proxy knightcrawler-addon:7000
}

2
deployment/docker/optional_reverse_proxy/configs/snippets/cloudflare-replace-X-Forwarded-For Normal file
View File

@@ -0,0 +1,2 @@
header_up X-Real-IP {http.request.header.CF-Connecting-IP}
header_up X-Forwarded-For {http.request.header.CF-Connecting-IP}

39
deployment/docker/optional_reverse_proxy/docker-compose.yaml Normal file
View File

@@ -0,0 +1,39 @@
name: caddy
networks:
caddy:
name: caddy
external: true
# cloudflare-tunnel:
# name: cloudflare-tunnel
# external: true
volumes:
config:
data:
services:
# Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS
# https://github.com/caddyserver/caddy
caddy:
cap_add:
- NET_ADMIN
container_name: caddy
hostname: caddy
image: caddy:2.7.6 # May be out of date, but using the `:latest` tag is how you accidentally break your server
networks:
- caddy
# - cloudflare-tunnel
ports:
- 80:80
- 443:443
- 443:443/udp
restart: always
security_opt:
- no-new-privileges:true
volumes:
- ./configs/:/etc/caddy/ # /etc/caddy/Caddyfile and /etc/caddy/snippets/
- ./logs:/var/log/caddy/
- config:/config
- data:/data

0
deployment/docker/optional_reverse_proxy/logs/.gitkeep Normal file
View File