Commit Graph

21 Commits

Author SHA1 Message Date
funkecoder23
2593bbdb73 update readme with more info
add info about installing docker

update DMM setup with correct path for env

reorder env setup and DMM setup

add info about pgloader and combine SQL command command into a docker exec

use github markdown for notes and important info
2024-02-04 20:29:06 -05:00
iPromKnight
b773ffcb20 Logging expansion
Adds Promtail and Loki to the metrics stack, configuring them as a datasource in grafana on deployment
Adds a dashboard to monitor the system (logs for producer and consumer)
Restructures the base directory so deployment manifests are moved into a deployments folder
Adds selective labels to producer, consumer and addon so that their logs are ingested by promtail
2024-02-04 19:50:28 +00:00
iPromKnight
7ba38db0cc remove .env and add .env.example - add readme instructions 2024-02-04 15:45:32 +00:00
Gabisonfire
5fe3f471e9 Renames project to Knight Crawler 2024-02-03 23:45:06 -05:00
HDwayne
45e17799b2 Add monitoring for Postgres metrics in README 2024-02-04 01:29:55 +01:00
HDwayne
679a658463 Fix docker-compose command in README 2024-02-03 17:17:13 +01:00
HDwayne
8e991488ce Add monitoring capabilities with Grafana and Prometheus 2024-02-03 17:09:10 +01:00
David Young
679bba3030 Minimal docs on importing rarbg dump
Signed-off-by: David Young <davidy@funkypenguin.co.nz>
2024-02-03 23:07:12 +13:00
purple_emily
1a71d618bd Add a simple to do list 2024-02-02 15:40:12 +00:00
purple_emily
428bdff413 Added a section on how to generate a GitHub PAT 2024-02-02 15:37:01 +00:00
purple_emily
2ddd30f621 Add an overview section to the README 2024-02-02 15:04:00 +00:00
iPromKnight
ab17ef81be Big rewrite - distributed consumers for ingestion / scraping(scalable) - single producer written in c#.
Changed from page scraping to rss xml scraping
Includes RealDebridManager hashlist decoding (requires a github readonly PAT as requests must be authenticated) - This allows ingestion of 200k+ entries in a few hours.
Simplifies a lot of torrentio to deal with new data
2024-02-01 16:38:45 +00:00
iPromKnight
da997af64a Move compose to top level
Comment out broken scrapers

Remove db init scripts and use ENABLE_SYNC env var which the code base uses to specify if Sequelize should create or alter tables on startup, meaning we dont need manual db initialization. There were missing tables anyhow :P
2024-01-18 08:54:57 +00:00
Gabisonfire
a34e40fbc8 First self-hosted release 2024-01-17 00:27:03 -05:00
TheBeastLT
cef5d33d84 fi readme 2022-06-17 22:07:23 +03:00
TheBeastLT
cf83b4f17b adjust enabled sscrapers 2022-06-17 07:13:56 +03:00
TheBeastLT
a253e8e0b0 remove scraper 2022-06-16 17:43:27 +03:00
TheBeastLT
58aba322c2 moves scraper to a subpackage 2020-03-10 15:23:52 +01:00
TheBeastLT
2618694e33 adds rarbg dump scraper based on scraped imdb ids 2020-03-09 15:23:16 +01:00
TheBeastLT
0f91c98b84 refactors scrapers and add kat and unofficial tpb dump scraper 2020-02-23 23:18:58 +01:00
TheBeastLT
5a122683d7 Initial commit 2019-03-13 22:41:15 +01:00