tools/knightcrawler
1
0
Fork 0
mirror of https://github.com/knightcrawler-stremio/knightcrawler.git synced 2024-12-20 03:29:51 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,055 Commits 7 Branches 31 Tags
a9bfc3d583c93158d4328f91e878e3b1d3eebdfe
Commit Graph

1055 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
iPromKnight
eef3265850 Merge pull request #55 from Gabisonfire/critical-security-enhancements 
Critical security enhancements
 2024-02-04 16:04:42 +00:00
iPromKnight
2b715b90a7 Remove postgres exporter port bind 2024-02-04 16:02:47 +00:00
purple_emily
170d126fc9 Formatting 2024-02-04 16:02:47 +00:00
purple_emily
f4df6e27a7 Close the ports for RabbitMQ 2024-02-04 16:02:47 +00:00
purple_emily
562713f749 Close the MongoDB port 2024-02-04 16:02:23 +00:00
purple_emily
050c22e11e Change wording of security comment 2024-02-04 15:58:10 +00:00
purple_emily
bc2c36b92f Close the external PostgreSQL port 2024-02-04 15:58:10 +00:00
iPromKnight
875d79b44d Merge pull request #46 from purple-emily/change-to-single-env-file 
Simplify the environment variables
 2024-02-04 15:56:25 +00:00
iPromKnight
7ba38db0cc remove .env and add .env.example - add readme instructions 2024-02-04 15:45:32 +00:00
iPromKnight
e461e26b0f Change postgres configuration in the producer to use the env vars from the stack 2024-02-04 15:03:07 +00:00
purple_emily
c20a92e8f0 Remove old environment files 2024-02-04 08:34:49 +00:00
purple_emily
cd3c2d3fe6 Merge and simplify the environment variables in 
addon and consumer.

Todo: producer

Change DATABASE_URI to be generic POSTGRES variables

DOES NOT WORK - First pass at upgrading environment variables

PostgreSQL environment variables have been split for addon and consumer. ENABLE_SYNC hard coded as `true`

MongoDB variables update.

Make the addon code more similar to the consumer code

Get some parity between addon and consumer
 2024-02-04 08:30:55 +00:00
purple_emily
b7c3c4376b Merge pull request #51 from Gabisonfire/feat/renaming 
Renames project to Knight Crawler
 2024-02-04 08:30:24 +00:00
Gabisonfire
5fe3f471e9 Renames project to Knight Crawler 2024-02-03 23:45:06 -05:00
Gabisonfire
a8173a0ba7 Merge pull request #48 from HDwayne/master 
Monitoring with Grafana and Prometheus
 2024-02-03 19:42:49 -05:00
Gabisonfire
adcf911d10 Merge pull request #49 from iPromKnight/master 
Package updates and start of logging
 2024-02-03 19:38:52 -05:00
HDwayne
45e17799b2 Add monitoring for Postgres metrics in README 2024-02-04 01:29:55 +01:00
HDwayne
5f691ef877 Add Postgres exporter service 2024-02-04 01:18:34 +01:00
iPromKnight
d660a72d6f missed a couple of log lines for logger 2024-02-03 22:12:45 +00:00
iPromKnight
8c1b6fc91b roll logging levels out throughout the consumer 2024-02-03 22:06:38 +00:00
iPromKnight
4bd76d682f remove pm2 process manager - let the container orchestrator handle the lifetime events (docker, podman, k8s etc) 2024-02-03 22:06:02 +00:00
iPromKnight
678b3857f3 Add pino as a logger, and update cache manager 
Rewrites the cache service to use the latest implementation of cache manager, and bring in the new mongo package.
Introduce Logger service
Removes bluebird as a dependency
 2024-02-03 22:05:15 +00:00
HDwayne
a56b04f04b Add Temporary statistics endpoint 2024-02-03 22:06:13 +01:00
HDwayne
679a658463 Fix docker-compose command in README 2024-02-03 17:17:13 +01:00
HDwayne
8e991488ce Add monitoring capabilities with Grafana and Prometheus 2024-02-03 17:09:10 +01:00
Gabisonfire
f9746af9cd Merge pull request #47 from geek-cookbook/add-rarbg-docs 
Minimal docs on importing rarbg dump
 2024-02-03 07:43:00 -05:00
David Young
679bba3030 Minimal docs on importing rarbg dump 
Signed-off-by: David Young <davidy@funkypenguin.co.nz>
 2024-02-03 23:07:12 +13:00
Gabisonfire
cb0b22e6a9 Merge pull request #40 from purple-emily/update-the-documentation 
Update the documentation
 2024-02-02 11:01:30 -05:00
Gabisonfire
898ab6ed1b Merge pull request #26 from iPromKnight/master 
Big rewrite of scraping solution
 2024-02-02 11:00:34 -05:00
purple_emily
1a71d618bd Add a simple to do list 2024-02-02 15:40:12 +00:00
purple_emily
428bdff413 Added a section on how to generate a GitHub PAT 2024-02-02 15:37:01 +00:00
Gabisonfire
fd03760155 Add issues template 2024-02-02 10:13:00 -05:00
purple_emily
2ddd30f621 Add an overview section to the README 2024-02-02 15:04:00 +00:00
iPromKnight
57f4757541 Implement Max Queue and Max Batch size when publishing 
MaxPublishBatchSize must be set, but MaxQueueSize can be set to 0 to disable check of the rabbitmq queue size
 2024-02-02 14:43:29 +00:00
iPromKnight
8ad6cf731c harden consumer 2024-02-02 14:04:53 +00:00
iPromKnight
68edaba308 Introduce max batch size, and configurable publish window 
Still need to implement queue size limit
Also fixes env var consistency between addon and consumer
 2024-02-02 13:49:54 +00:00
iPromKnight
0644fe7720 add the jackett backend alternate addon "jackettio" 2024-02-02 13:27:41 +00:00
iPromKnight
188ffd10f3 rework removing providers filters, and clean up project a little 
Also wraps in pm2, and introduces linting, and dev watch
 2024-02-02 13:27:15 +00:00
iPromKnight
c1169a15ee Add editor config 2024-02-02 13:25:57 +00:00
iPromKnight
ee994fc8be ignore bin and obj 2024-02-01 16:47:45 +00:00
iPromKnight
ab17ef81be Big rewrite - distributed consumers for ingestion / scraping(scalable) - single producer written in c#. 
Changed from page scraping to rss xml scraping
Includes RealDebridManager hashlist decoding (requires a github readonly PAT as requests must be authenticated) - This allows ingestion of 200k+ entries in a few hours.
Simplifies a lot of torrentio to deal with new data
 2024-02-01 16:38:45 +00:00
Gabisonfire
6fb4ddcf23 Merge pull request #20 from Gabisonfire/devel 
Sync with upstream
 2024-01-31 20:29:43 -05:00
Gabisonfire
b2bba2a8aa Sync with upstream 2024-01-31 19:53:47 -05:00
Gabisonfire
d63f1d9716 Merge pull request #19 from KillTrot/master 
Change 1337x FlareSolverr mechanic to only use FlareSolverr on the first request
 2024-01-31 16:12:44 -05:00
Cedric
a0017f3491 Merge branch 'Gabisonfire:master' into master 2024-01-31 15:00:28 -06:00
Cedric
3f6c95b12f Remove FlareSolverr Variables after scraping 2024-01-31 14:59:41 -06:00
Cedric
aa101758f0 Merge branch 'master' of https://github.com/KillTrot/torrentio-scraper-sh 2024-01-31 14:21:48 -06:00
Cedric
ee34b6db17 Change Flaresolverr mechanic to use cookies after initial flaresolverr request 2024-01-31 14:19:46 -06:00
TheBeastLT
fb267b8057 remove psql logging 2024-01-31 09:30:59 +02:00
TheBeastLT
79b83e52e4 reduce psql connection pool a bit 2024-01-31 09:25:33 +02:00