1015 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
funkecoder23	2d27926816	correct the psql command ... escaped quotes to properly parse the "createdAt" and "updatedAt" column names	2024-02-04 20:29:28 -05:00
funkecoder23	101b5c416e	Add optional configuration section ... Add details about consumer replicas Add CONSUMER_REPLICAS to .env.example Change `consumer: deploy: replicas` to use CONSUMER_REPLICAS var to keep user config options in one file	2024-02-04 20:29:28 -05:00
funkecoder23	2593bbdb73	update readme with more info ... add info about installing docker update DMM setup with correct path for env reorder env setup and DMM setup add info about pgloader and combine SQL command command into a docker exec use github markdown for notes and important info	2024-02-04 20:29:06 -05:00
Gabisonfire	24f35d10db	Merge pull request #64 from trulow/patch-2 ... Updated and replaced selfhostio entries with knightcrawler	2024-02-04 20:06:48 -05:00
Gabisonfire	469f643526	Merge pull request #58 from iPromKnight/promtail-loki ... Logging expansion	2024-02-04 20:03:09 -05:00
trulow	d4e82e9878	Updated and replaced selfhostio entries with knightcrawler ... Updated and replaced selfhostio entries with knightcrawler. Was previously still pointing to the old DB name of selfhostio	2024-02-04 16:52:05 -08:00
purple_emily	6e362cfb89	Merge pull request #62 from FunkeCoder23/hotfix-manifest-version	2024-02-04 22:30:15 +00:00
funkecoder23	3ddc922081	remove extraneous v from manifest version	2024-02-04 17:19:11 -05:00
iPromKnight	dae52c1bc4	forgot to update paths - doh	2024-02-04 20:32:08 +00:00
iPromKnight	b773ffcb20	Logging expansion ... Adds Promtail and Loki to the metrics stack, configuring them as a datasource in grafana on deployment Adds a dashboard to monitor the system (logs for producer and consumer) Restructures the base directory so deployment manifests are moved into a deployments folder Adds selective labels to producer, consumer and addon so that their logs are ingested by promtail	2024-02-04 19:50:28 +00:00
iPromKnight	eef3265850	Merge pull request #55 from Gabisonfire/critical-security-enhancements ... Critical security enhancements	2024-02-04 16:04:42 +00:00
iPromKnight	2b715b90a7	Remove postgres exporter port bind	2024-02-04 16:02:47 +00:00
purple_emily	170d126fc9	Formatting	2024-02-04 16:02:47 +00:00
purple_emily	f4df6e27a7	Close the ports for RabbitMQ	2024-02-04 16:02:47 +00:00
purple_emily	562713f749	Close the MongoDB port	2024-02-04 16:02:23 +00:00
purple_emily	050c22e11e	Change wording of security comment	2024-02-04 15:58:10 +00:00
purple_emily	bc2c36b92f	Close the external PostgreSQL port	2024-02-04 15:58:10 +00:00
iPromKnight	875d79b44d	Merge pull request #46 from purple-emily/change-to-single-env-file ... Simplify the environment variables	2024-02-04 15:56:25 +00:00
iPromKnight	7ba38db0cc	remove .env and add .env.example - add readme instructions	2024-02-04 15:45:32 +00:00
iPromKnight	e461e26b0f	Change postgres configuration in the producer to use the env vars from the stack	2024-02-04 15:03:07 +00:00
purple_emily	c20a92e8f0	Remove old environment files	2024-02-04 08:34:49 +00:00
purple_emily	cd3c2d3fe6	Merge and simplify the environment variables in ... addon and consumer. Todo: producer Change DATABASE_URI to be generic POSTGRES variables DOES NOT WORK - First pass at upgrading environment variables PostgreSQL environment variables have been split for addon and consumer. ENABLE_SYNC hard coded as `true` MongoDB variables update. Make the addon code more similar to the consumer code Get some parity between addon and consumer	2024-02-04 08:30:55 +00:00
purple_emily	b7c3c4376b	Merge pull request #51 from Gabisonfire/feat/renaming ... Renames project to Knight Crawler	2024-02-04 08:30:24 +00:00
Gabisonfire	5fe3f471e9	Renames project to Knight Crawler	2024-02-03 23:45:06 -05:00
Gabisonfire	a8173a0ba7	Merge pull request #48 from HDwayne/master ... Monitoring with Grafana and Prometheus	2024-02-03 19:42:49 -05:00
Gabisonfire	adcf911d10	Merge pull request #49 from iPromKnight/master ... Package updates and start of logging	2024-02-03 19:38:52 -05:00
HDwayne	45e17799b2	Add monitoring for Postgres metrics in README	2024-02-04 01:29:55 +01:00
HDwayne	5f691ef877	Add Postgres exporter service	2024-02-04 01:18:34 +01:00
iPromKnight	d660a72d6f	missed a couple of log lines for logger	2024-02-03 22:12:45 +00:00
iPromKnight	8c1b6fc91b	roll logging levels out throughout the consumer	2024-02-03 22:06:38 +00:00
iPromKnight	4bd76d682f	remove pm2 process manager - let the container orchestrator handle the lifetime events (docker, podman, k8s etc)	2024-02-03 22:06:02 +00:00
iPromKnight	678b3857f3	Add pino as a logger, and update cache manager ... Rewrites the cache service to use the latest implementation of cache manager, and bring in the new mongo package. Introduce Logger service Removes bluebird as a dependency	2024-02-03 22:05:15 +00:00
HDwayne	a56b04f04b	Add Temporary statistics endpoint	2024-02-03 22:06:13 +01:00
HDwayne	679a658463	Fix docker-compose command in README	2024-02-03 17:17:13 +01:00
HDwayne	8e991488ce	Add monitoring capabilities with Grafana and Prometheus	2024-02-03 17:09:10 +01:00
Gabisonfire	f9746af9cd	Merge pull request #47 from geek-cookbook/add-rarbg-docs ... Minimal docs on importing rarbg dump	2024-02-03 07:43:00 -05:00
David Young	679bba3030	Minimal docs on importing rarbg dump ... Signed-off-by: David Young <davidy@funkypenguin.co.nz>	2024-02-03 23:07:12 +13:00
Gabisonfire	cb0b22e6a9	Merge pull request #40 from purple-emily/update-the-documentation ... Update the documentation	2024-02-02 11:01:30 -05:00
Gabisonfire	898ab6ed1b	Merge pull request #26 from iPromKnight/master ... Big rewrite of scraping solution	2024-02-02 11:00:34 -05:00
purple_emily	1a71d618bd	Add a simple to do list	2024-02-02 15:40:12 +00:00
purple_emily	428bdff413	Added a section on how to generate a GitHub PAT	2024-02-02 15:37:01 +00:00
Gabisonfire	fd03760155	Add issues template	2024-02-02 10:13:00 -05:00
purple_emily	2ddd30f621	Add an overview section to the README	2024-02-02 15:04:00 +00:00
iPromKnight	57f4757541	Implement Max Queue and Max Batch size when publishing ... MaxPublishBatchSize must be set, but MaxQueueSize can be set to 0 to disable check of the rabbitmq queue size	2024-02-02 14:43:29 +00:00
iPromKnight	8ad6cf731c	harden consumer	2024-02-02 14:04:53 +00:00
iPromKnight	68edaba308	Introduce max batch size, and configurable publish window ... Still need to implement queue size limit Also fixes env var consistency between addon and consumer	2024-02-02 13:49:54 +00:00
iPromKnight	0644fe7720	add the jackett backend alternate addon "jackettio"	2024-02-02 13:27:41 +00:00
iPromKnight	188ffd10f3	rework removing providers filters, and clean up project a little ... Also wraps in pm2, and introduces linting, and dev watch	2024-02-02 13:27:15 +00:00
iPromKnight	c1169a15ee	Add editor config	2024-02-02 13:25:57 +00:00
iPromKnight	ee994fc8be	ignore bin and obj	2024-02-01 16:47:45 +00:00