Producer / Consumer / Collector rewrite (#160)

* Converted metadata service to redis

* move to postgres instead

* fix global usings

* [skip ci] optimize wolverine by prebuilding static types

* [skip ci] Stop indexing mac folder indexes

* [skip ci] producer, metadata and migrations

removed mongodb
added redis cache
imdb meta in postgres
Enable pgtrm
Create trigrams index
Add search meta postgres function

* [skip ci] get rid of node folder, replace mongo with redis in consumer

also wire up postgres metadata searches

* [skip ci] change mongo to redis in the addon

* [skip ci] jackettio to redis

* Rest of mongo removed...

* Cleaner rerunning of metadata - without conflicts

* Add akas import as well as basic metadata

* Include episodes file too

* cascade truncate pre-import

* reverse order to avoid cascadeing

* separate out clean to separate handler

* Switch producer to use metadata matching pre-preocessing dmm

* More work

* Still porting PTN

* PTN port, adding tests

* [skip ci] Codec tests

* [skip ci] Complete Collection handler tests

* [skip ci] container tests

* [skip ci] Convert handlers tests

* [skip ci] DateHandler tests

* [skip ci] Dual Audio matching tests

* [skip ci] episode code tests

* [skip ci] Extended handler tests

* [skip ci] group handler tests

* [skip ci] some broken stuff right now

* [skip ci] more ptn

* [skip ci] PTN now in a separate nuget package, rebased this on the redis changes - i need them.

* [skip ci] Wire up PTN port. Tired - will test tomorrow

* [skip ci] Needs a lot of work - too many titles being missed now

* cleaner. done?

* Handle the date in the imdb search

- add integer function to confirm its a valid integer
- use the input date as a range of -+1 year

* [skip ci] Start of collector service for RD

[skip ci] WIP

Implemented metadata saga, along with channels to process up to a maximum of 100 infohashes each time
The saga will rety for each infohas by requeuing up to three times, before just marking as complete for that infoHash - meaning no data will be updated in the db for that torrent.

[skip ci] Ready to test with queue publishing

Will provision a fanout exchange if it doesn't exist, and create and bind a queue to it. Listens to the queue with 50 prefetch count.
Still needs PTN rewrite bringing in to parse the filename response from real debrid, and extract season and episode numbers if the file is a tvshow

[skip ci] Add Debrid Collector Build Job

Debrid Collector ready for testing

New consumer, new collector, producer has meta lookup and anti porn measures

[skip ci] WIP - moving from wolverine to MassTransit.

 not happy that wolverine cannot effectively control saga concurrency. we need to really.

[skip ci] Producer and new Consumer moved to MassTransit

Just the debrid collector to go now, then to write the optional qbit collector.

Collector now switched to mass transit too

hide porn titles in logs, clean up cache name in redis for imdb titles

[skip ci] Allow control of queues

[skip ci] Update deployment

Remove old consumer, fix deployment files, fix dockerfiles for shared project import

fix base deployment

* Add collector missing env var

* edits to kick off builds

* Add optional qbit deployment which qbit collector will use

* Qbit collector done

* reorder compose, and bring both qbit and qbitcollector into the compose, with 0 replicas as default

* Clean up compose file

* Ensure debrid collector errors if no debrid api key
This commit is contained in:
iPromKnight
2024-03-25 23:32:28 +00:00
committed by GitHub
parent 9c6c1ac249
commit 9a831e92d0
443 changed files with 4154 additions and 476262 deletions

5
src/addon/.dockerignore Normal file
View File

@@ -0,0 +1,5 @@
**/node_modules
**/npm-debug.log
**/.env
.eslint*
Dockerfile

1
src/addon/.eslintignore Normal file
View File

@@ -0,0 +1 @@
*.ts

39
src/addon/.eslintrc.cjs Normal file
View File

@@ -0,0 +1,39 @@
/** @type {import("eslint").ESLint.Options} */
module.exports = {
env: {
es2024: true,
node: true,
},
globals: {
Atomics: "readonly",
SharedArrayBuffer: "readonly",
},
parserOptions: {
sourceType: "module",
},
plugins: ["import", "import-helpers"],
rules: {
"default-case": "off",
"import/no-duplicates": "off",
"import/no-extraneous-dependencies": ["off", { devDependencies: ["backend", "frontend", "mobile"] }],
"import/order": "off",
"import-helpers/order-imports": [
"warn",
{
alphabetize: {
order: "asc",
},
},
],
"lines-between-class-members": ["error", "always", { exceptAfterSingleLine: true }],
"no-continue": "off",
"no-param-reassign": "off",
"no-plusplus": ["error", { allowForLoopAfterthoughts: true }],
"no-restricted-syntax": "off",
"no-unused-expressions": ["off", { allowShortCircuit: true }],
"no-unused-vars": "off",
"no-use-before-define": "off",
"one-var": ["error", { uninitialized: "consecutive" }],
"prefer-destructuring": "warn",
},
};

35
src/addon/Dockerfile Normal file
View File

@@ -0,0 +1,35 @@
# --- Build Stage ---
FROM node:lts-alpine AS builder
RUN apk update && apk upgrade && \
apk add --no-cache git
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build
# --- Runtime Stage ---
FROM node:lts-alpine
# Install pm2
RUN npm install pm2 -g
WORKDIR /app
ENV NODE_ENV production
COPY --from=builder /app ./
RUN npm prune --omit=dev
# CIS-DI-0001
RUN addgroup -S addon && adduser -S -G addon addon
USER addon
EXPOSE 7000
ENTRYPOINT [ "pm2-runtime", "start", "ecosystem.config.cjs"]
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD pm2 ping || exit 1

View File

@@ -0,0 +1,14 @@
module.exports = {
apps: [
{
name: "knightcrawler",
script: "npm start",
cwd: "/app",
watch: ["./dist/index.cjs"],
autorestart: true,
env: {
...process.env
},
},
],
};

68
src/addon/esbuild.js Normal file
View File

@@ -0,0 +1,68 @@
import { build } from "esbuild";
import { copy } from 'esbuild-plugin-copy';
import { readFileSync, rmSync } from "fs";
const { devDependencies } = JSON.parse(readFileSync("./package.json", "utf8"));
const start = Date.now();
try {
const outdir = "dist";
rmSync(outdir, { recursive: true, force: true });
build({
bundle: true,
entryPoints: [
"./src/index.js",
// "./src/**/*.css",
// "./src/**/*.hbs",
// "./src/**/*.html"
],
external: [...(devDependencies && Object.keys(devDependencies))],
keepNames: true,
loader: {
".css": "copy",
".hbs": "copy",
".html": "copy",
},
minify: true,
outbase: "./src",
outdir,
outExtension: {
".js": ".cjs",
},
platform: "node",
plugins: [
{
name: "populate-import-meta",
setup: ({ onLoad }) => {
onLoad({ filter: new RegExp(`${import.meta.dirname}/src/.*.(js|ts)$`) }, args => {
const contents = readFileSync(args.path, "utf8");
const transformedContents = contents
.replace(/import\.meta/g, `{dirname:__dirname,filename:__filename}`)
.replace(/import\.meta\.filename/g, "__filename")
.replace(/import\.meta\.dirname/g, "__dirname");
return { contents: transformedContents, loader: "default" };
});
},
},
copy({
assets: [
{
from: ['./static/**'],
to: ['./static'],
},
],
})
],
}).then(() => {
// biome-ignore lint/style/useTemplate: <explanation>
console.log("⚡ " + "\x1b[32m" + `Done in ${Date.now() - start}ms`);
});
} catch (e) {
console.log(e);
process.exit(1);
}

21
src/addon/jsconfig.json Normal file
View File

@@ -0,0 +1,21 @@
{
"compilerOptions": {
"baseUrl": "./src",
"checkJs": true,
"isolatedModules": true,
"lib": ["es6"],
"module": "ESNext",
"moduleResolution": "node",
"outDir": "./dist",
"pretty": true,
"removeComments": true,
"resolveJsonModule": true,
"rootDir": "./src",
"skipLibCheck": true,
"sourceMap": true,
"target": "ES6",
"types": ["node"],
"typeRoots": ["node_modules/@types", "src/@types"]
},
"exclude": ["node_modules"]
}

5551
src/addon/package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

53
src/addon/package.json Normal file
View File

@@ -0,0 +1,53 @@
{
"name": "knightcrawler-addon",
"version": "0.0.1",
"type": "module",
"scripts": {
"build": "node esbuild.js",
"dev": "tsx watch --ignore node_modules src/index.js",
"start": "node dist/index.cjs",
"lint": "eslint . --ext .ts,.js"
},
"dependencies": {
"@putdotio/api-client": "^8.42.0",
"all-debrid-api": "^1.1.0",
"axios": "^1.6.1",
"bottleneck": "^2.19.5",
"cache-manager": "^3.4.4",
"cors": "^2.8.5",
"debrid-link-api": "^1.0.1",
"express-rate-limit": "^6.7.0",
"ip": "^1.1.8",
"magnet-uri": "^6.2.0",
"name-to-imdb": "^3.0.4",
"named-queue": "^2.2.1",
"offcloud-api": "^1.0.2",
"parse-torrent-title": "git://github.com/TheBeastLT/parse-torrent-title.git#022408972c2a040f846331a912a6a8487746a654",
"pg": "^8.11.3",
"pg-hstore": "^2.3.4",
"premiumize-api": "^1.0.3",
"prom-client": "^12.0.0",
"real-debrid-api": "git://github.com/TheBeastLT/node-real-debrid.git#d1f7eaa8593b947edbfbc8a92a176448b48ef445",
"request-ip": "^3.3.0",
"router": "^1.3.8",
"sequelize": "^6.31.1",
"stremio-addon-sdk": "^1.6.10",
"swagger-stats": "^0.99.7",
"ua-parser-js": "^1.0.36",
"user-agents": "^1.0.1444",
"@redis/client": "^1.5.14",
"@redis/json": "^1.0.6",
"@redis/search": "^1.1.6",
"cache-manager-redis-store": "^2.0.0"
},
"devDependencies": {
"@types/node": "^20.11.6",
"@types/stremio-addon-sdk": "^1.6.10",
"esbuild": "^0.19.12",
"esbuild-plugin-copy": "^2.1.1",
"eslint": "^8.56.0",
"eslint-plugin-import": "^2.29.1",
"eslint-plugin-import-helpers": "^1.3.1",
"tsx": "^4.7.0"
}
}

123
src/addon/src/addon.js Normal file
View File

@@ -0,0 +1,123 @@
import Bottleneck from 'bottleneck';
import { addonBuilder } from 'stremio-addon-sdk';
import { cacheWrapStream } from './lib/cache.js';
import { dummyManifest } from './lib/manifest.js';
import * as repository from './lib/repository.js';
import applySorting from './lib/sort.js';
import { toStreamInfo, applyStaticInfo } from './lib/streamInfo.js';
import { Type } from './lib/types.js';
import { applyMochs, getMochCatalog, getMochItemMeta } from './moch/moch.js';
import StaticLinks from './moch/static.js';
const CACHE_MAX_AGE = parseInt(process.env.CACHE_MAX_AGE) || 60 * 60; // 1 hour in seconds
const CACHE_MAX_AGE_EMPTY = 60; // 60 seconds
const CATALOG_CACHE_MAX_AGE = 20 * 60; // 20 minutes
const STALE_REVALIDATE_AGE = 4 * 60 * 60; // 4 hours
const STALE_ERROR_AGE = 7 * 24 * 60 * 60; // 7 days
const builder = new addonBuilder(dummyManifest());
const limiter = new Bottleneck({
maxConcurrent: process.env.LIMIT_MAX_CONCURRENT || 200,
highWater: process.env.LIMIT_QUEUE_SIZE || 220,
strategy: Bottleneck.strategy.OVERFLOW
});
builder.defineStreamHandler((args) => {
if (!args.id.match(/tt\d+/i) && !args.id.match(/kitsu:\d+/i)) {
return Promise.resolve({ streams: [] });
}
return cacheWrapStream(args.id, () => limiter.schedule(() =>
streamHandler(args)
.then(records => records
.sort((a, b) => b.torrent.seeders - a.torrent.seeders || b.torrent.uploadDate - a.torrent.uploadDate)
.map(record => toStreamInfo(record)))))
.then(streams => applySorting(streams, args.extra))
.then(streams => applyStaticInfo(streams))
.then(streams => applyMochs(streams, args.extra))
.then(streams => enrichCacheParams(streams))
.catch(error => {
console.log(`Failed request ${args.id}: ${error}`);
return Promise.reject(`Failed request ${args.id}: ${error}`);
});
});
builder.defineCatalogHandler((args) => {
const mochKey = args.id.replace("knightcrawler-", '');
console.log(`Incoming catalog ${args.id} request with skip=${args.extra.skip || 0}`)
return getMochCatalog(mochKey, args.extra)
.then(metas => ({
metas: metas,
cacheMaxAge: CATALOG_CACHE_MAX_AGE
}))
.catch(error => {
return Promise.reject(`Failed retrieving catalog ${args.id}: ${JSON.stringify(error)}`);
});
})
builder.defineMetaHandler((args) => {
const [mochKey, metaId] = args.id.split(':');
console.log(`Incoming debrid meta ${args.id} request`)
return getMochItemMeta(mochKey, metaId, args.extra)
.then(meta => ({
meta: meta,
cacheMaxAge: metaId === 'Downloads' ? 0 : CACHE_MAX_AGE
}))
.catch(error => {
return Promise.reject(`Failed retrieving catalog meta ${args.id}: ${JSON.stringify(error)}`);
});
})
async function streamHandler(args) {
if (args.type === Type.MOVIE) {
return movieRecordsHandler(args);
} else if (args.type === Type.SERIES) {
return seriesRecordsHandler(args);
}
return Promise.reject('not supported type');
}
async function seriesRecordsHandler(args) {
if (args.id.match(/^tt\d+:\d+:\d+$/)) {
const [imdbId, season = "1", episode = "1"] = args.id.split(':');
const parsedSeason = parseInt(season, 10);
const parsedEpisode = parseInt(episode, 10);
return repository.getImdbIdSeriesEntries(imdbId, parsedSeason, parsedEpisode);
} else if (args.id.match(/^kitsu:\d+(?::\d+)?$/i)) {
const [, kitsuId, episodePart] = args.id.split(':');
const episode = episodePart !== undefined ? parseInt(episodePart, 10) : undefined;
return episode !== undefined
? repository.getKitsuIdSeriesEntries(kitsuId, episode)
: repository.getKitsuIdMovieEntries(kitsuId);
}
return Promise.resolve([]);
}
async function movieRecordsHandler(args) {
if (args.id.match(/^tt\d+$/)) {
const [imdbId] = args.id.split(':');
console.log("imdbId", imdbId);
return repository.getImdbIdMovieEntries(imdbId);
} else if (args.id.match(/^kitsu:\d+(?::\d+)?$/i)) {
return seriesRecordsHandler(args);
}
return Promise.resolve([]);
}
function enrichCacheParams(streams) {
let cacheAge = CACHE_MAX_AGE;
if (!streams.length) {
cacheAge = CACHE_MAX_AGE_EMPTY;
} else if (streams.every(stream => stream?.url?.endsWith(StaticLinks.FAILED_ACCESS))) {
cacheAge = 0;
}
return {
streams: streams,
cacheMaxAge: cacheAge,
staleRevalidate: STALE_REVALIDATE_AGE,
staleError: STALE_ERROR_AGE
}
}
export default builder.getInterface();

13
src/addon/src/index.js Normal file
View File

@@ -0,0 +1,13 @@
import express from 'express';
import { initBestTrackers } from './lib/magnetHelper.js';
import serverless from './serverless.js';
const app = express();
app.enable('trust proxy');
app.use(express.static('static', { maxAge: '1y' }));
app.use((req, res, next) => serverless(req, res, next));
app.listen(process.env.PORT || 7000, () => {
initBestTrackers()
.then(() => console.log(`Started addon at: http://localhost:${process.env.PORT || 7000}`));
});

View File

@@ -0,0 +1,93 @@
import cacheManager from 'cache-manager';
import { cacheConfig } from './config.js';
import { isStaticUrl } from '../moch/static.js';
import redisStore from "cache-manager-redis-store";
const GLOBAL_KEY_PREFIX = 'knightcrawler-addon';
const STREAM_KEY_PREFIX = `${GLOBAL_KEY_PREFIX}|stream`;
const AVAILABILITY_KEY_PREFIX = `${GLOBAL_KEY_PREFIX}|availability`;
const RESOLVED_URL_KEY_PREFIX = `${GLOBAL_KEY_PREFIX}|resolved`;
const STREAM_TTL = process.env.STREAM_TTL || 60 * 60; // 1 Hour
const STREAM_EMPTY_TTL = process.env.STREAM_EMPTY_TTL || 30; // 1 minute
const AVAILABILITY_TTL = 8 * 60 * 60; // 8 hours
const AVAILABILITY_EMPTY_TTL = 30 * 60; // 30 minutes
const MESSAGE_VIDEO_URL_TTL = 60; // 1 minutes
// When the streams are empty we want to cache it for less time in case of timeouts or failures
const memoryCache = initiateMemoryCache();
const remoteCache = initiateRemoteCache();
function initiateRemoteCache() {
if (cacheConfig.NO_CACHE) {
return null;
} else if (cacheConfig.REDIS_CONNECTION_STRING) {
return cacheManager.caching({
store: redisStore,
ttl: STREAM_EMPTY_TTL,
url: cacheConfig.REDIS_CONNECTION_STRING
});
} else {
return cacheManager.caching({
store: 'memory',
ttl: STREAM_EMPTY_TTL
});
}
}
function initiateMemoryCache() {
return cacheManager.caching({
store: 'memory',
ttl: MESSAGE_VIDEO_URL_TTL,
max: Infinity // infinite LRU cache size
});
}
function cacheWrap(cache, key, method, options) {
if (cacheConfig.NO_CACHE || !cache) {
return method();
}
return cache.wrap(key, method, options);
}
export function cacheWrapStream(id, method) {
return cacheWrap(remoteCache, `${STREAM_KEY_PREFIX}:${id}`, method, {
ttl: (streams) => streams.length ? STREAM_TTL : STREAM_EMPTY_TTL
});
}
export function cacheWrapResolvedUrl(id, method) {
return cacheWrap(memoryCache, `${RESOLVED_URL_KEY_PREFIX}:${id}`, method, {
ttl: (url) => isStaticUrl(url) ? MESSAGE_VIDEO_URL_TTL : STREAM_TTL
});
}
export function cacheAvailabilityResults(results) {
Object.keys(results)
.forEach(infoHash => {
const key = `${AVAILABILITY_KEY_PREFIX}:${infoHash}`;
const value = results[infoHash];
const ttl = value?.length ? AVAILABILITY_TTL : AVAILABILITY_EMPTY_TTL;
memoryCache.set(key, value, { ttl })
});
return results;
}
export function getCachedAvailabilityResults(infoHashes) {
const keys = infoHashes.map(infoHash => `${AVAILABILITY_KEY_PREFIX}:${infoHash}`)
return new Promise(resolve => {
memoryCache.mget(...keys, (error, result) => {
if (error) {
console.log('Failed retrieve availability cache', error)
return resolve({});
}
const availabilityResults = {};
infoHashes.forEach((infoHash, index) => {
if (result[index]) {
availabilityResults[infoHash] = result[index];
}
});
resolve(availabilityResults);
})
});
}

View File

@@ -0,0 +1,28 @@
export const cacheConfig = {
REDIS_CONNECTION_STRING: process.env.REDIS_CONNECTION_STRING || 'redis://localhost:6379/0',
NO_CACHE: parseBool(process.env.NO_CACHE, false),
}
export const databaseConfig = {
POSTGRES_HOST: process.env.POSTGRES_HOST || 'postgres',
POSTGRES_PORT: process.env.POSTGRES_PORT || '5432',
POSTGRES_DB: process.env.POSTGRES_DB || 'knightcrawler',
POSTGRES_USER: process.env.POSTGRES_USER || 'postgres',
POSTGRES_PASSWORD: process.env.POSTGRES_PASSWORD || 'postgres',
}
// Combine the environment variables into a connection string
// The combined string will look something like:
// 'postgres://postgres:postgres@localhost:5432/knightcrawler'
databaseConfig.POSTGRES_URI = 'postgres://' + databaseConfig.POSTGRES_USER + ':' + databaseConfig.POSTGRES_PASSWORD + '@' + databaseConfig.POSTGRES_HOST + ':' + databaseConfig.POSTGRES_PORT + '/' + databaseConfig.POSTGRES_DB;
function parseBool(boolString, defaultValue) {
const isString = typeof boolString === 'string' || boolString instanceof String;
if (!isString) {
return defaultValue;
}
return boolString.toLowerCase() === 'true' ? true : defaultValue;
}

View File

@@ -0,0 +1,28 @@
import { DebridOptions } from '../moch/options.js';
import { QualityFilter, SizeFilter } from './filter.js';
import { LanguageOptions } from './languages.js';
const keysToSplit = [LanguageOptions.key, QualityFilter.key, SizeFilter.key, DebridOptions.key];
const keysToUppercase = [SizeFilter.key];
export function parseConfiguration(configuration) {
if (!configuration) {
return undefined;
}
const configValues = configuration.split('|')
.reduce((map, next) => {
const [key, value] = next.split('=');
if (key && value) {
map[key.toLowerCase()] = value;
}
return map;
}, {});
keysToSplit
.filter(key => configValues[key])
.forEach(key => configValues[key] = configValues[key].split(',')
.map(value => keysToUppercase.includes(key) ? value.toUpperCase() : value.toLowerCase()))
return configValues;
}

View File

@@ -0,0 +1,72 @@
const VIDEO_EXTENSIONS = [
"3g2",
"3gp",
"avi",
"flv",
"mkv",
"mk3d",
"mov",
"mp2",
"mp4",
"m4v",
"mpe",
"mpeg",
"mpg",
"mpv",
"webm",
"wmv",
"ogm",
"ts",
"m2ts"
];
const SUBTITLE_EXTENSIONS = [
"aqt",
"gsub",
"jss",
"sub",
"ttxt",
"pjs",
"psb",
"rt",
"smi",
"slt",
"ssf",
"srt",
"ssa",
"ass",
"usf",
"idx",
"vtt"
];
const DISK_EXTENSIONS = [
"iso",
"m2ts",
"ts",
"vob"
]
const ARCHIVE_EXTENSIONS = [
"rar",
"zip"
]
export function isVideo(filename) {
return isExtension(filename, VIDEO_EXTENSIONS);
}
export function isSubtitle(filename) {
return isExtension(filename, SUBTITLE_EXTENSIONS);
}
export function isDisk(filename) {
return isExtension(filename, DISK_EXTENSIONS);
}
export function isArchive(filename) {
return isExtension(filename, ARCHIVE_EXTENSIONS);
}
export function isExtension(filename, extensions) {
const extensionMatch = filename?.match(/\.(\w{2,4})$/);
return extensionMatch && extensions.includes(extensionMatch[1].toLowerCase());
}

130
src/addon/src/lib/filter.js Normal file
View File

@@ -0,0 +1,130 @@
import { parseSize, extractSize } from './titleHelper.js';
import { Type } from './types.js';
export const QualityFilter = {
key: 'qualityfilter',
options: [
{
key: 'brremux',
label: 'BluRay REMUX',
test(quality, bingeGroup) {
return bingeGroup?.includes(this.label);
}
},
{
key: 'hdrall',
label: 'HDR/HDR10+/Dolby Vision',
items: ['HDR', 'HDR10+', 'DV'],
test(quality) {
const hdrProfiles = quality?.split(' ')?.slice(1)?.join() || '';
return this.items.some(hdrType => hdrProfiles.includes(hdrType));
}
},
{
key: 'dolbyvision',
label: 'Dolby Vision',
test(quality) {
const hdrProfiles = quality?.split(' ')?.slice(1)?.join() || '';
return hdrProfiles === 'DV';
}
},
{
key: '4k',
label: '4k',
items: ['4k'],
test(quality) {
return quality && this.items.includes(quality.split(' ')[0]);
}
},
{
key: '1080p',
label: '1080p',
items: ['1080p'],
test(quality) {
return this.items.includes(quality)
}
},
{
key: '720p',
label: '720p',
items: ['720p'],
test(quality) {
return this.items.includes(quality)
}
},
{
key: '480p',
label: '480p',
items: ['480p'],
test(quality) {
return this.items.includes(quality)
}
},
{
key: 'other',
label: 'Other (DVDRip/HDRip/BDRip...)',
// could be ['DVDRip', 'HDRip', 'BDRip', 'BRRip', 'BluRay', 'WEB-DL', 'WEBRip', 'HDTV', 'DivX', 'XviD']
items: ['4k', '1080p', '720p', '480p', 'SCR', 'CAM', 'TeleSync', 'TeleCine'],
test(quality) {
return quality && !this.items.includes(quality.split(' ')[0]);
}
},
{
key: 'scr',
label: 'Screener',
items: ['SCR'],
test(quality) {
return this.items.includes(quality)
}
},
{
key: 'cam',
label: 'Cam',
items: ['CAM', 'TeleSync', 'TeleCine'],
test(quality) {
return this.items.includes(quality)
}
},
{
key: 'unknown',
label: 'Unknown',
test(quality) {
return !quality
}
}
]
};
export const SizeFilter = {
key: 'sizefilter'
}
export default function applyFilters(streams, config) {
return [
filterByQuality,
filterBySize
].reduce((filteredStreams, filter) => filter(filteredStreams, config), streams);
}
function filterByQuality(streams, config) {
const filters = config[QualityFilter.key];
if (!filters) {
return streams;
}
const filterOptions = QualityFilter.options.filter(option => filters.includes(option.key));
return streams.filter(stream => {
const [ , streamQuality] = stream.name.split('\n');
const bingeGroup = stream.behaviorHints?.bingeGroup;
return !filterOptions.some(option => option.test(streamQuality, bingeGroup));
});
}
function filterBySize(streams, config) {
const sizeFilters = config[SizeFilter.key];
if (!sizeFilters?.length) {
return streams;
}
const sizeLimit = parseSize(config.type === Type.MOVIE ? sizeFilters.shift() : sizeFilters.pop());
return streams.filter(stream => {
const size = extractSize(stream.title)
return size <= sizeLimit;
})
}

View File

@@ -0,0 +1,466 @@
const STYLESHEET = `
* {
box-sizing: border-box;
}
body,
html {
margin: 0;
padding: 0;
width: 100%;
height: 100%
}
html {
background-size: auto 100%;
background-size: cover;
background-position: center center;
background-repeat: repeat-y;
}
body {
display: flex;
background-color: transparent;
font-family: 'Open Sans', Arial, sans-serif;
color: white;
}
h1 {
font-size: 4.5vh;
font-weight: 700;
}
h2 {
font-size: 2.2vh;
font-weight: normal;
font-style: italic;
opacity: 0.8;
}
h3 {
font-size: 2.2vh;
}
h1,
h2,
h3,
p,
label {
margin: 0;
text-shadow: 0 0 1vh rgba(0, 0, 0, 0.15);
}
p {
font-size: 1.75vh;
}
ul {
font-size: 1.75vh;
margin: 0;
margin-top: 1vh;
padding-left: 3vh;
}
a {
color: green
}
a.install-link {
text-decoration: none
}
.install-button {
border: 0;
outline: 0;
color: white;
background: #8A5AAB;
padding: 1.2vh 3.5vh;
margin: auto;
text-align: center;
font-family: 'Open Sans', Arial, sans-serif;
font-size: 2.2vh;
font-weight: 600;
cursor: pointer;
display: block;
box-shadow: 0 0.5vh 1vh rgba(0, 0, 0, 0.2);
transition: box-shadow 0.1s ease-in-out;
}
.install-button:hover {
box-shadow: none;
}
.install-button:active {
box-shadow: 0 0 0 0.5vh white inset;
}
#addon {
width: 90vh;
margin: auto;
padding-left: 10%;
padding-right: 10%;
background: rgba(0, 0, 0, 0.60);
}
.logo {
height: 14vh;
width: 14vh;
margin: auto;
margin-bottom: 3vh;
}
.logo img {
width: 100%;
}
.name, .version {
display: inline-block;
vertical-align: top;
}
.name {
line-height: 5vh;
}
.version {
position: absolute;
line-height: 5vh;
margin-left: 1vh;
opacity: 0.8;
}
.contact {
left: 0;
bottom: 4vh;
width: 100%;
margin-top: 1vh;
text-align: center;
}
.contact a {
font-size: 1.4vh;
font-style: italic;
}
.separator {
margin-bottom: 4vh;
}
.label {
font-size: 2.2vh;
font-weight: 600;
padding: 0;
line-height: inherit;
}
.btn-group, .multiselect-container {
width: 100%;
}
.btn {
text-align: left;
}
.multiselect-container {
border: 0;
border-radius: 0;
}
.input, .btn {
width: 100%;
margin: auto;
margin-bottom: 10px;
padding: 6px 12px;
border: 0;
border-radius: 0;
outline: 0;
color: #333;
background-color: rgb(255, 255, 255);
box-shadow: 0 0.5vh 1vh rgba(0, 0, 0, 0.2);
}
.input:focus, .btn:focus {
outline: none;
box-shadow: 0 0 0 2pt rgb(30, 144, 255, 0.7);
}
`;
import { MochOptions } from '../moch/moch.js';
import { DebridOptions } from '../moch/options.js';
import { QualityFilter, SizeFilter } from './filter.js';
import { LanguageOptions } from './languages.js';
import { SortOptions } from './sort.js';
export default function landingTemplate(manifest, config = {}) {
const sort = config[SortOptions.key] || SortOptions.options.qualitySeeders.key;
const languages = config[LanguageOptions.key] || [];
const qualityFilters = config[QualityFilter.key] || [];
const sizeFilter = (config[SizeFilter.key] || []).join(',');
const limit = config.limit || '';
const debridProvider = Object.keys(MochOptions).find(mochKey => config[mochKey]);
const debridOptions = config[DebridOptions.key] || [];
const realDebridApiKey = config[MochOptions.realdebrid.key] || '';
const premiumizeApiKey = config[MochOptions.premiumize.key] || '';
const allDebridApiKey = config[MochOptions.alldebrid.key] || '';
const debridLinkApiKey = config[MochOptions.debridlink.key] || '';
const offcloudApiKey = config[MochOptions.offcloud.key] || '';
const putioKey = config[MochOptions.putio.key] || '';
const putioClientId = putioKey.replace(/@.*/, '');
const putioToken = putioKey.replace(/.*@/, '');
const background = manifest.background || 'https://dl.strem.io/addon-background.jpg';
const logo = manifest.logo || 'https://dl.strem.io/addon-logo.png';
const sortOptionsHTML = Object.values(SortOptions.options)
.map((option, i) => `<option value="${option.key}" ${i === 0 ? 'selected' : ''}>${option.description}</option>`)
.join('\n');
const languagesOptionsHTML = LanguageOptions.options
.map((option, i) => `<option value="${option.key}">${option.label}</option>`)
.join('\n');
const qualityFiltersHTML = Object.values(QualityFilter.options)
.map(option => `<option value="${option.key}">${option.label}</option>`)
.join('\n');
const debridProvidersHTML = Object.values(MochOptions)
.map(moch => `<option value="${moch.key}">${moch.name}</option>`)
.join('\n');
const debridOptionsHTML = Object.values(DebridOptions.options)
.map(option => `<option value="${option.key}">${option.description}</option>`)
.join('\n');
const stylizedTypes = manifest.types
.map(t => t[0].toUpperCase() + t.slice(1) + (t !== 'series' ? 's' : ''));
return `
<!DOCTYPE html>
<html style="background-image: url(${background});">
<head>
<meta charset="utf-8">
<title>${manifest.name} - Stremio Addon</title>
<link rel="shortcut icon" href="${logo}" type="image/x-icon">
<link href="https://fonts.googleapis.com/css?family=Open+Sans:400,600,700&display=swap" rel="stylesheet">
<script src="https://code.jquery.com/jquery-3.6.4.slim.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/popper.js@1.16.1/dist/umd/popper.min.js"></script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/js/bootstrap.min.js"></script>
<link href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css" rel="stylesheet" >
<script src="https://cdnjs.cloudflare.com/ajax/libs/bootstrap-multiselect/1.1.2/js/bootstrap-multiselect.min.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/bootstrap-multiselect/1.1.2/css/bootstrap-multiselect.css" rel="stylesheet"/>
<style>${STYLESHEET}</style>
</head>
<body>
<div id="addon">
<div class="logo">
<img src="${logo}">
</div>
<h1 class="name">${manifest.name}</h1>
<h2 class="version">${manifest.version || '0.0.0'}</h2>
<h2 class="description">${manifest.description || ''}</h2>
<div class="separator"></div>
<h3 class="gives">This addon has more :</h3>
<ul>
${stylizedTypes.map(t => `<li>${t}</li>`).join('')}
</ul>
<div class="separator"></div>
<label class="label" for="iSort">Sorting:</label>
<select id="iSort" class="input" onchange="sortModeChange()">
${sortOptionsHTML}
</select>
<label class="label" for="iLanguages">Priority foreign language:</label>
<select id="iLanguages" class="input" onchange="generateInstallLink()" name="languages[]" multiple="multiple" title="Streams with the selected dubs/subs language will be shown on the top">
${languagesOptionsHTML}
</select>
<label class="label" for="iQualityFilter">Exclude qualities/resolutions:</label>
<select id="iQualityFilter" class="input" onchange="generateInstallLink()" name="qualityFilters[]" multiple="multiple">
${qualityFiltersHTML}
</select>
<label class="label" id="iLimitLabel" for="iLimit">Max results per quality:</label>
<input type="text" inputmode="numeric" pattern="[0-9]*" id="iLimit" onchange="generateInstallLink()" class="input" placeholder="All results">
<label class="label" id="iSizeFilterLabel" for="iSizeFilter">Video size limit:</label>
<input type="text" pattern="([0-9.]*(?:MB|GB),?)+" id="iSizeFilter" onchange="generateInstallLink()" class="input" placeholder="No limit" title="Returned videos cannot exceed this size, use comma to have different size for movies and series. Examples: 5GB ; 800MB ; 10GB,2GB">
<label class="label" for="iDebridProviders">Debrid provider:</label>
<select id="iDebridProviders" class="input" onchange="debridProvidersChange()">
<option value="none" selected>None</option>
${debridProvidersHTML}
</select>
<div id="dRealDebrid">
<label class="label" for="iRealDebrid">RealDebrid API Key (Find it <a href='https://real-debrid.com/apitoken' target="_blank">here</a>):</label>
<input type="text" id="iRealDebrid" onchange="generateInstallLink()" class="input">
</div>
<div id="dAllDebrid">
<label class="label" for="iAllDebrid">AllDebrid API Key (Create it <a href='https://alldebrid.com/apikeys' target="_blank">here</a>):</label>
<input type="text" id="iAllDebrid" onchange="generateInstallLink()" class="input">
</div>
<div id="dPremiumize">
<label class="label" for="iPremiumize">Premiumize API Key (Find it <a href='https://www.premiumize.me/account' target="_blank">here</a>):</label>
<input type="text" id="iPremiumize" onchange="generateInstallLink()" class="input">
</div>
<div id="dDebridLink">
<label class="label" for="iDebridLink">DebridLink API Key (Find it <a href='https://debrid-link.fr/webapp/apikey' target="_blank">here</a>):</label>
<input type="text" id="iDebridLink" onchange="generateInstallLink()" class="input">
</div>
<div id="dOffcloud">
<label class="label" for="iOffcloud">Offcloud API Key (Find it <a href='https://offcloud.com/#/account' target="_blank">here</a>):</label>
<input type="text" id="iOffcloud" onchange="generateInstallLink()" class="input">
</div>
<div id="dPutio">
<label class="label" for="iPutio">Put.io ClientId and Token (Create new OAuth App <a href='https://app.put.io/oauth' target="_blank">here</a>):</label>
<input type="text" id="iPutioClientId" placeholder="ClientId" onchange="generateInstallLink()" class="input">
<input type="text" id="iPutioToken" placeholder="Token" onchange="generateInstallLink()" class="input">
</div>
<div id="dDebridOptions">
<label class="label" for="iDebridOptions">Debrid options:</label>
<select id="iDebridOptions" class="input" onchange="generateInstallLink()" name="debridOptions[]" multiple="multiple">
${debridOptionsHTML}
</select>
</div>
<div class="separator"></div>
<a id="installLink" class="install-link" href="#">
<button name="Install" class="install-button">INSTALL</button>
</a>
<div class="contact">
<p>Or paste into Stremio search bar after clicking install</p>
</div>
<div class="separator"></div>
</div>
<script type="text/javascript">
$(document).ready(function() {
const isTvMedia = window.matchMedia("tv").matches;
const isTvAgent = /\\b(?:tv|wv)\\b/i.test(navigator.userAgent)
const isDesktopMedia = window.matchMedia("(pointer:fine)").matches;
if (isDesktopMedia && !isTvMedia && !isTvAgent) {
$('#iLanguages').multiselect({
nonSelectedText: 'None',
buttonTextAlignment: 'left',
onChange: () => generateInstallLink()
});
$('#iLanguages').multiselect('select', [${languages.map(language => '"' + language + '"')}]);
$('#iQualityFilter').multiselect({
nonSelectedText: 'None',
buttonTextAlignment: 'left',
onChange: () => generateInstallLink()
});
$('#iQualityFilter').multiselect('select', [${qualityFilters.map(filter => '"' + filter + '"')}]);
$('#iDebridOptions').multiselect({
nonSelectedText: 'None',
buttonTextAlignment: 'left',
onChange: () => generateInstallLink()
});
$('#iDebridOptions').multiselect('select', [${debridOptions.map(option => '"' + option + '"')}]);
} else {
$('#iLanguages').val([${languages.map(language => '"' + language + '"')}]);
$('#iQualityFilter').val([${qualityFilters.map(filter => '"' + filter + '"')}]);
$('#iDebridOptions').val([${debridOptions.map(option => '"' + option + '"')}]);
}
$('#iDebridProviders').val("${debridProvider || 'none'}");
$('#iRealDebrid').val("${realDebridApiKey}");
$('#iPremiumize').val("${premiumizeApiKey}");
$('#iAllDebrid').val("${allDebridApiKey}");
$('#iDebridLink').val("${debridLinkApiKey}");
$('#iOffcloud').val("${offcloudApiKey}");
$('#iPutioClientId').val("${putioClientId}");
$('#iPutioToken').val("${putioToken}");
$('#iSort').val("${sort}");
$('#iLimit').val("${limit}");
$('#iSizeFilter').val("${sizeFilter}");
generateInstallLink();
debridProvidersChange();
});
function sortModeChange() {
if (['${SortOptions.options.seeders.key}', '${SortOptions.options.size.key}'].includes($('#iSort').val())) {
$("#iLimitLabel").text("Max results:");
} else {
$("#iLimitLabel").text("Max results per quality:");
}
generateInstallLink();
}
function debridProvidersChange() {
const provider = $('#iDebridProviders').val()
$('#dDebridOptions').toggle(provider !== 'none');
$('#dRealDebrid').toggle(provider === '${MochOptions.realdebrid.key}');
$('#dPremiumize').toggle(provider === '${MochOptions.premiumize.key}');
$('#dAllDebrid').toggle(provider === '${MochOptions.alldebrid.key}');
$('#dDebridLink').toggle(provider === '${MochOptions.debridlink.key}');
$('#dOffcloud').toggle(provider === '${MochOptions.offcloud.key}');
$('#dPutio').toggle(provider === '${MochOptions.putio.key}');
}
function generateInstallLink() {
const qualityFilterValue = $('#iQualityFilter').val().join(',') || '';
const sortValue = $('#iSort').val() || '';
const languagesValue = $('#iLanguages').val().join(',') || [];
const limitValue = $('#iLimit').val() || '';
const sizeFilterValue = $('#iSizeFilter').val() || '';
const debridOptionsValue = $('#iDebridOptions').val().join(',') || '';
const realDebridValue = $('#iRealDebrid').val() || '';
const allDebridValue = $('#iAllDebrid').val() || '';
const debridLinkValue = $('#iDebridLink').val() || ''
const premiumizeValue = $('#iPremiumize').val() || '';
const offcloudValue = $('#iOffcloud').val() || ''
const putioClientIdValue = $('#iPutioClientId').val() || '';
const putioTokenValue = $('#iPutioToken').val() || '';
const qualityFilters = qualityFilterValue.length && qualityFilterValue;
const sort = sortValue !== '${SortOptions.options.qualitySeeders.key}' && sortValue;
const languages = languagesValue.length && languagesValue;
const limit = /^[1-9][0-9]{0,2}$/.test(limitValue) && limitValue;
const sizeFilter = sizeFilterValue.length && sizeFilterValue;
const debridOptions = debridOptionsValue.length && debridOptionsValue.trim();
const realDebrid = realDebridValue.length && realDebridValue.trim();
const premiumize = premiumizeValue.length && premiumizeValue.trim();
const allDebrid = allDebridValue.length && allDebridValue.trim();
const debridLink = debridLinkValue.length && debridLinkValue.trim();
const offcloud = offcloudValue.length && offcloudValue.trim();
const putio = putioClientIdValue.length && putioTokenValue.length && putioClientIdValue.trim() + '@' + putioTokenValue.trim();
let configurationValue = [
['${SortOptions.key}', sort],
['${LanguageOptions.key}', languages],
['${QualityFilter.key}', qualityFilters],
['limit', limit],
['${SizeFilter.key}', sizeFilter],
['${DebridOptions.key}', debridOptions],
['${MochOptions.realdebrid.key}', realDebrid],
['${MochOptions.premiumize.key}', premiumize],
['${MochOptions.alldebrid.key}', allDebrid],
['${MochOptions.debridlink.key}', debridLink],
['${MochOptions.offcloud.key}', offcloud],
['${MochOptions.putio.key}', putio]
].filter(([_, value]) => value.length).map(([key, value]) => key + '=' + value).join('|');
const configuration = configurationValue && configurationValue.length ? '/' + configurationValue : '';
const location = window.location.host + configuration + '/manifest.json'
installLink.href = 'stremio://' + location;
console.log("Install link: " + installLink.href.replace('stremio://', 'https://'));
}
installLink.addEventListener('click', function() {
navigator.clipboard.writeText(installLink.href.replace('stremio://', 'https://'));
});
</script>
</body>
</html>`
}

View File

@@ -0,0 +1,76 @@
const languageMapping = {
'dubbed': 'Dubbed',
'multi audio': 'Multi Audio',
'multi subs': 'Multi Subs',
'dual audio': 'Dual Audio',
'english': '🇬🇧',
'japanese': '🇯🇵',
'russian': '🇷🇺',
'italian': '🇮🇹',
'portuguese': '🇵🇹',
'spanish': '🇪🇸',
'latino': '🇲🇽',
'korean': '🇰🇷',
'chinese': '🇨🇳',
'taiwanese': '🇹🇼',
'french': '🇫🇷',
'german': '🇩🇪',
'dutch': '🇳🇱',
'hindi': '🇮🇳',
'telugu': '🇮🇳',
'tamil': '🇮🇳',
'polish': '🇵🇱',
'lithuanian': '🇱🇹',
'latvian': '🇱🇻',
'estonian': '🇪🇪',
'czech': '🇨🇿',
'slovakian': '🇸🇰',
'slovenian': '🇸🇮',
'hungarian': '🇭🇺',
'romanian': '🇷🇴',
'bulgarian': '🇧🇬',
'serbian': '🇷🇸 ',
'croatian': '🇭🇷',
'ukrainian': '🇺🇦',
'greek': '🇬🇷',
'danish': '🇩🇰',
'finnish': '🇫🇮',
'swedish': '🇸🇪',
'norwegian': '🇳🇴',
'turkish': '🇹🇷',
'arabic': '🇸🇦',
'persian': '🇮🇷',
'hebrew': '🇮🇱',
'vietnamese': '🇻🇳',
'indonesian': '🇮🇩',
'malay': '🇲🇾',
'thai': '🇹🇭'
}
export const LanguageOptions = {
key: 'language',
options: Object.keys(languageMapping).slice(5).map(lang => ({
key: lang,
label: `${languageMapping[lang]} ${lang.charAt(0).toUpperCase()}${lang.slice(1)}`
}))
}
export function mapLanguages(languages) {
const mapped = languages
.map(language => languageMapping[language])
.filter(language => language)
.sort((a, b) => Object.values(languageMapping).indexOf(a) - Object.values(languageMapping).indexOf(b));
const unmapped = languages
.filter(language => !languageMapping[language])
.sort((a, b) => a.localeCompare(b))
return [...new Set([].concat(mapped).concat(unmapped))];
}
export function containsLanguage(stream, languages) {
return languages.map(lang => languageMapping[lang]).some(lang => stream.title.includes(lang));
}
export function languageFromCode(code) {
const entry = Object.entries(languageMapping).find(entry => entry[1] === code);
return entry?.[0];
}

View File

@@ -0,0 +1,88 @@
import axios from 'axios';
import magnet from 'magnet-uri';
import { getTorrent } from './repository.js';
import { getRandomUserAgent } from './requestHelper.js';
import { Type } from './types.js';
const TRACKERS_URL = 'https://raw.githubusercontent.com/ngosang/trackerslist/master/trackers_best.txt';
const DEFAULT_TRACKERS = [
"udp://47.ip-51-68-199.eu:6969/announce",
"udp://9.rarbg.me:2940",
"udp://9.rarbg.to:2820",
"udp://exodus.desync.com:6969/announce",
"udp://explodie.org:6969/announce",
"udp://ipv4.tracker.harry.lu:80/announce",
"udp://open.stealth.si:80/announce",
"udp://opentor.org:2710/announce",
"udp://opentracker.i2p.rocks:6969/announce",
"udp://retracker.lanta-net.ru:2710/announce",
"udp://tracker.cyberia.is:6969/announce",
"udp://tracker.dler.org:6969/announce",
"udp://tracker.ds.is:6969/announce",
"udp://tracker.internetwarriors.net:1337",
"udp://tracker.openbittorrent.com:6969/announce",
"udp://tracker.opentrackr.org:1337/announce",
"udp://tracker.tiny-vps.com:6969/announce",
"udp://tracker.torrent.eu.org:451/announce",
"udp://valakas.rollo.dnsabr.com:2710/announce",
"udp://www.torrent.eu.org:451/announce",
]
const ANIME_TRACKERS = [
"http://nyaa.tracker.wf:7777/announce",
"http://anidex.moe:6969/announce",
"http://tracker.anirena.com:80/announce",
"udp://tracker.uw0.xyz:6969/announce",
"http://share.camoe.cn:8080/announce",
"http://t.nyaatracker.com:80/announce",
];
const RUSSIAN_TRACKERS = [
"udp://opentor.net:6969",
"http://bt.t-ru.org/ann?magnet",
"http://bt2.t-ru.org/ann?magnet",
"http://bt3.t-ru.org/ann?magnet",
"http://bt4.t-ru.org/ann?magnet",
];
let BEST_TRACKERS = [];
let ALL_ANIME_TRACKERS = [];
let ALL_RUSSIAN_TRACKERS = [];
export async function getMagnetLink(infoHash) {
const torrent = await getTorrent(infoHash).catch(() => ({ infoHash }));
const torrentTrackers = torrent?.trackers?.split(',') || [];
const animeTrackers = torrent.type === Type.ANIME ? ALL_ANIME_TRACKERS : [];
const trackers = unique([].concat(torrentTrackers).concat(animeTrackers));
return magnet.encode({ infoHash: infoHash, name: torrent.title, announce: trackers });
}
export async function initBestTrackers() {
BEST_TRACKERS = await getBestTrackers();
ALL_ANIME_TRACKERS = unique(BEST_TRACKERS.concat(DEFAULT_TRACKERS).concat(ANIME_TRACKERS));
ALL_RUSSIAN_TRACKERS = unique(BEST_TRACKERS.concat(DEFAULT_TRACKERS).concat(RUSSIAN_TRACKERS));
}
async function getBestTrackers(retry = 2) {
const options = { timeout: 30000, headers: { 'User-Agent': getRandomUserAgent() } };
return axios.get(TRACKERS_URL, options)
.then(response => response?.data?.trim()?.split('\n\n') || [])
.catch(error => {
if (retry === 0) {
console.log(`Failed retrieving best trackers: ${error.message}`);
throw error;
}
return getBestTrackers(retry - 1);
});
}
export function getSources(trackersInput, infoHash) {
if (!trackersInput) {
return null;
}
const trackers = Array.isArray(trackersInput) ? trackersInput : trackersInput.split(',');
return trackers.map(tracker => `tracker:${tracker}`).concat(`dht:${infoHash}`);
}
function unique(array) {
return Array.from(new Set(array));
}

View File

@@ -0,0 +1,71 @@
import { MochOptions } from '../moch/moch.js';
import { showDebridCatalog } from '../moch/options.js';
import { Type } from './types.js';
const CatalogMochs = Object.values(MochOptions).filter(moch => moch.catalog);
export function manifest(config = {}) {
return {
id: 'com.stremio.knightcrawler.knightcrawler',
version: '0.0.1',
name: getName(config),
description: getDescription(),
catalogs: getCatalogs(config),
resources: getResources(config),
types: [Type.MOVIE, Type.SERIES, Type.ANIME, Type.OTHER],
background: "https://i.ibb.co/PTzsRZ0/kc-final-e2x-bg.png",
logo: "https://i.ibb.co/fQxLJ1C/kc-final-e.png",
behaviorHints: {
configurable: true,
configurationRequired: false,
}
};
}
export function dummyManifest() {
const manifestDefault = manifest();
manifestDefault.catalogs = [{ id: 'dummy', type: Type.OTHER }];
manifestDefault.resources = ['stream', 'meta'];
return manifestDefault;
}
function getName(config) {
const rootName = 'Knight Crawler';
const mochSuffix = Object.values(MochOptions)
.filter(moch => config[moch.key])
.map(moch => moch.shortName)
.join('/');
return [rootName, mochSuffix].filter(v => v).join(' ');
}
function getDescription() {
return 'Selfhost the Torrentio brings you much Funio';
}
function getCatalogs(config) {
return CatalogMochs
.filter(moch => showDebridCatalog(config) && config[moch.key])
.map(moch => ({
id: `knightcrawler-${moch.key}`,
name: `${moch.name}`,
type: 'other',
extra: [{ name: 'skip' }],
}));
}
function getResources(config) {
const streamResource = {
name: 'stream',
types: [Type.MOVIE, Type.SERIES],
idPrefixes: ['tt', 'kitsu']
};
const metaResource = {
name: 'meta',
types: [Type.OTHER],
idPrefixes: CatalogMochs.filter(moch => config[moch.key]).map(moch => moch.key)
};
if (showDebridCatalog(config) && CatalogMochs.filter(moch => config[moch.key]).length) {
return [streamResource, metaResource];
}
return [streamResource];
}

View File

@@ -0,0 +1,20 @@
/**
* Delay promise
*/
export async function delay(duration) {
return new Promise((resolve) => setTimeout(resolve, duration));
}
/**
* Timeout promise after a set time in ms
*/
export async function timeout(timeoutMs, promise, message = 'Timed out') {
return Promise.race([
promise,
new Promise(function (resolve, reject) {
setTimeout(function () {
reject(message);
}, timeoutMs);
})
]);
}

View File

@@ -0,0 +1,165 @@
import { Sequelize } from 'sequelize';
import { databaseConfig } from './config.js';
const { Op } = Sequelize;
const database = new Sequelize(
databaseConfig.POSTGRES_URI,
{
logging: false
}
);
const Torrent = database.define('torrent',
{
infoHash: { type: Sequelize.STRING(64), primaryKey: true },
provider: { type: Sequelize.STRING(32), allowNull: false },
torrentId: { type: Sequelize.STRING(128) },
title: { type: Sequelize.STRING(256), allowNull: false },
size: { type: Sequelize.BIGINT },
type: { type: Sequelize.STRING(16), allowNull: false },
uploadDate: { type: Sequelize.DATE, allowNull: false },
seeders: { type: Sequelize.SMALLINT },
trackers: { type: Sequelize.STRING(4096) },
languages: { type: Sequelize.STRING(4096) },
resolution: { type: Sequelize.STRING(16) }
}
);
const File = database.define('file',
{
id: { type: Sequelize.BIGINT, autoIncrement: true, primaryKey: true },
infoHash: {
type: Sequelize.STRING(64),
allowNull: false,
references: { model: Torrent, key: 'infoHash' },
onDelete: 'CASCADE'
},
fileIndex: { type: Sequelize.INTEGER },
title: { type: Sequelize.STRING(256), allowNull: false },
size: { type: Sequelize.BIGINT },
imdbId: { type: Sequelize.STRING(32) },
imdbSeason: { type: Sequelize.INTEGER },
imdbEpisode: { type: Sequelize.INTEGER },
kitsuId: { type: Sequelize.INTEGER },
kitsuEpisode: { type: Sequelize.INTEGER }
},
);
const Subtitle = database.define('subtitle',
{
infoHash: {
type: Sequelize.STRING(64),
allowNull: false,
references: { model: Torrent, key: 'infoHash' },
onDelete: 'CASCADE'
},
fileIndex: { type: Sequelize.INTEGER, allowNull: false },
fileId: {
type: Sequelize.BIGINT,
allowNull: true,
references: { model: File, key: 'id' },
onDelete: 'SET NULL'
},
title: { type: Sequelize.STRING(512), allowNull: false },
size: { type: Sequelize.BIGINT, allowNull: false },
},
{ timestamps: false }
);
Torrent.hasMany(File, { foreignKey: 'infoHash', constraints: false });
File.belongsTo(Torrent, { foreignKey: 'infoHash', constraints: false });
File.hasMany(Subtitle, { foreignKey: 'fileId', constraints: false });
Subtitle.belongsTo(File, { foreignKey: 'fileId', constraints: false });
export function getTorrent(infoHash) {
return Torrent.findOne({ where: { infoHash: infoHash } });
}
export function getFiles(infoHashes) {
return File.findAll({ where: { infoHash: { [Op.in]: infoHashes } } });
}
export function getImdbIdMovieEntries(imdbId) {
return File.findAll({
where: {
imdbId: { [Op.eq]: imdbId }
},
include: [Torrent],
limit: 500,
order: [
[Torrent, 'size', 'DESC']
]
});
}
export function getImdbIdSeriesEntries(imdbId, season, episode) {
return File.findAll({
where: {
imdbId: { [Op.eq]: imdbId },
imdbSeason: { [Op.eq]: season },
imdbEpisode: { [Op.eq]: episode }
},
include: [Torrent],
limit: 500,
order: [
[Torrent, 'size', 'DESC']
]
});
}
export function getKitsuIdMovieEntries(kitsuId) {
return File.findAll({
where: {
kitsuId: { [Op.eq]: kitsuId }
},
include: [Torrent],
limit: 500,
order: [
[Torrent, 'size', 'DESC']
]
});
}
export function getKitsuIdSeriesEntries(kitsuId, episode) {
return File.findAll({
where: {
kitsuId: { [Op.eq]: kitsuId },
kitsuEpisode: { [Op.eq]: episode }
},
include: [Torrent],
limit: 500,
order: [
[Torrent, 'size', 'DESC']
]
});
}
export async function computeTorrentStatistics() {
try {
// Global Stats
const totalTorrents = await Torrent.count();
// Stats by Provider
const torrentsByProvider = await Torrent.findAll({
attributes: ['provider', [Sequelize.fn('COUNT', Sequelize.col('provider')), 'count']],
group: ['provider']
});
// Stats by Content Type (Movie, Series, etc.)
const torrentsByType = await Torrent.findAll({
attributes: ['type', [Sequelize.fn('COUNT', Sequelize.col('type')), 'count']],
group: ['type']
});
return {
totalTorrents,
torrentsByProvider,
torrentsByType,
};
} catch (error) {
console.error('Error computing statistics:', error);
throw error;
}
}

View File

@@ -0,0 +1,6 @@
import UserAgent from 'user-agents';
const userAgent = new UserAgent();
export function getRandomUserAgent() {
return userAgent.random().toString();
}

50
src/addon/src/lib/sort.js Normal file
View File

@@ -0,0 +1,50 @@
import { containsLanguage, LanguageOptions } from './languages.js';
import { extractSize } from './titleHelper.js';
export const SortOptions = {
key: 'sort',
options: {
qualitySeeders: {
key: 'quality',
description: 'By quality then seeders'
},
qualitySize: {
key: 'qualitysize',
description: 'By quality then size'
},
seeders: {
key: 'seeders',
description: 'By seeders'
},
size: {
key: 'size',
description: 'By size'
},
}
}
export default function sortStreams(streams, config) {
const languages = config[LanguageOptions.key];
if (languages?.length && languages[0] !== 'english') {
// No need to filter english since it's hard to predict which entries are english
const streamsWithLanguage = streams.filter(stream => containsLanguage(stream, languages));
const streamsNoLanguage = streams.filter(stream => !streamsWithLanguage.includes(stream));
return _sortStreams(streamsWithLanguage, config).concat(_sortStreams(streamsNoLanguage, config));
}
return _sortStreams(streams, config);
}
function _sortStreams(streams, config) {
const limit = /^[1-9][0-9]*$/.test(config.limit) && parseInt(config.limit) || undefined;
return sortBySize(streams, limit);
}
function sortBySize(streams, limit) {
return streams
.sort((a, b) => {
const aSize = extractSize(a.title);
const bSize = extractSize(b.title);
return bSize - aSize;
}).slice(0, limit);
}

View File

@@ -0,0 +1,148 @@
import titleParser from 'parse-torrent-title';
import { mapLanguages } from './languages.js';
import { getSources } from './magnetHelper.js';
import { getSubtitles } from './subtitles.js';
import { Type } from './types.js';
const ADDON_NAME = 'knightcrawler';
const SIZE_DELTA = 0.02;
const UNKNOWN_SIZE = 300000000;
const CAM_SOURCES = ['CAM', 'TeleSync', 'TeleCine', 'SCR'];
export function toStreamInfo(record) {
const torrentInfo = titleParser.parse(record.torrent.title);
const fileInfo = titleParser.parse(record.title);
const sameInfo = !Number.isInteger(record.fileIndex)
|| Math.abs(record.size / record.torrent.size - 1) < SIZE_DELTA
|| record.title.includes(record.torrent.title);
const quality = getQuality(record, torrentInfo, fileInfo);
const hdrProfiles = torrentInfo.hdr || fileInfo.hdr || []
const title = joinDetailParts(
[
joinDetailParts([record.torrent.title.replace(/[, ]+/g, ' ')]),
joinDetailParts([!sameInfo && record.title || undefined]),
joinDetailParts([
joinDetailParts([formatSize(record.size)], '💾 ')
]),
joinDetailParts(getLanguages(record, torrentInfo, fileInfo), '', ' / '),
],
'',
'\n'
);
const name = joinDetailParts(
[
joinDetailParts([ADDON_NAME]),
joinDetailParts([quality, joinDetailParts(hdrProfiles, '', ' | ')])
],
'',
'\n'
);
const bingeGroupParts = getBingeGroupParts(record, sameInfo, quality, torrentInfo, fileInfo);
const bingeGroup = joinDetailParts(bingeGroupParts, "knightcrawler|", "|")
const behaviorHints = bingeGroup ? { bingeGroup } : undefined;
return cleanOutputObject({
name: name,
title: title,
infoHash: record.infoHash,
fileIdx: record.fileIndex,
behaviorHints: behaviorHints,
sources: getSources(record.torrent.trackers, record.infoHash),
subtitles: getSubtitles(record)
});
}
function getQuality(record, torrentInfo, fileInfo) {
if (CAM_SOURCES.includes(fileInfo.source)) {
return fileInfo.source;
}
if (CAM_SOURCES.includes(torrentInfo.source)) {
return torrentInfo.source;
}
const resolution = fileInfo.resolution || torrentInfo.resolution || record.torrent.resolution;
const source = fileInfo.source || torrentInfo.source;
return resolution || source;
}
function getLanguages(record, torrentInfo, fileInfo) {
const providerLanguages = record.torrent.languages && titleParser.parse(record.torrent.languages + '.srt').languages || [];
const torrentLanguages = torrentInfo.languages || [];
const fileLanguages = fileInfo.languages || [];
const dubbed = torrentInfo.dubbed || fileInfo.dubbed;
let languages = Array.from(new Set([].concat(torrentLanguages).concat(fileLanguages).concat(providerLanguages)));
if (record.kitsuId || record.torrent.type === Type.ANIME) {
// no need to display japanese for anime
languages = languages.concat(dubbed ? ['dubbed'] : [])
.filter(lang => lang !== 'japanese');
}
if (languages.length === 1 && languages.includes('english')) {
// no need to display languages if only english is present
languages = [];
}
if (languages.length === 0 && dubbed) {
// display dubbed only if there are no other languages defined for non anime
languages = ['dubbed'];
}
return mapLanguages(languages);
}
function joinDetailParts(parts, prefix = '', delimiter = ' ') {
const filtered = parts.filter((part) => part !== undefined && part !== null).join(delimiter);
return filtered.length > 0 ? `${prefix}${filtered}` : undefined;
}
function formatSize(size) {
if (!size) {
return undefined;
}
if (size === UNKNOWN_SIZE) {
return undefined;
}
const i = size === 0 ? 0 : Math.floor(Math.log(size) / Math.log(1024));
return Number((size / Math.pow(1024, i)).toFixed(2)) + ' ' + ['B', 'kB', 'MB', 'GB', 'TB'][i];
}
export function applyStaticInfo(streams) {
return streams.map(stream => enrichStaticInfo(stream));
}
function enrichStaticInfo(stream) {
return enrichSubtitles(stream);
}
function enrichSubtitles(stream) {
if (stream.subtitles?.length) {
stream.subtitles = stream.subtitles.map(subtitle =>{
if (subtitle.url) {
return subtitle;
}
return {
id: `${subtitle.fileIndex}`,
lang: subtitle.lang,
url: `http://localhost:11470/${subtitle.infoHash}/${subtitle.fileIndex}/${subtitle.title.split('/').pop()}`
};
});
}
return stream;
}
function getBingeGroupParts(record, sameInfo, quality, torrentInfo, fileInfo) {
if (record.torrent.type === Type.MOVIE) {
const source = torrentInfo.source || fileInfo.source
return [quality]
.concat(source !== quality ? source : [])
.concat(torrentInfo.codec || fileInfo.codec)
.concat(torrentInfo.bitDepth || fileInfo.bitDepth)
.concat(torrentInfo.hdr || fileInfo.hdr);
} else if (sameInfo) {
return [quality]
.concat(fileInfo.hdr)
.concat(fileInfo.group);
}
return [record.infoHash];
}
function cleanOutputObject(object) {
return Object.fromEntries(Object.entries(object).filter(([_, v]) => v != null));
}

View File

@@ -0,0 +1,92 @@
import { parse } from 'parse-torrent-title';
import { isExtension } from './extension.js';
const languageMapping = {
'english': 'eng',
'japanese': 'jpn',
'russian': 'rus',
'italian': 'ita',
'portuguese': 'por',
'spanish': 'spa',
'latino': 'lat',
'korean': 'kor',
'chinese': 'zho',
'taiwanese': 'zht',
'french': 'fre',
'german': 'ger',
'dutch': 'dut',
'hindi': 'hin ',
'telugu': 'tel',
'tamil': 'tam',
'polish': 'pol',
'lithuanian': 'lit',
'latvian': 'lav',
'estonian': 'est',
'czech': 'cze',
'slovakian': 'slo',
'slovenian': 'slv',
'hungarian': 'hun',
'romanian': 'rum',
'bulgarian': 'bul',
'serbian': 'scc',
'croatian': 'hrv',
'ukrainian': 'ukr',
'greek': 'ell',
'danish': 'dan',
'finnish': 'fin',
'swedish': 'swe',
'norwegian': 'nor',
'turkish': 'tur',
'arabic': 'ara',
'persian': 'per',
'hebrew': 'heb',
'vietnamese': 'vie',
'indonesian': 'ind',
'thai': 'tha'
}
const ignoreSet = new Set(['dubbed', 'multi audio', 'multi subs', 'dual audio']);
const allowedExtensions = ['srt', 'vtt', 'ass', 'ssa'];
export function getSubtitles(record) {
if (!record?.subtitles?.length) {
return null;
}
return record.subtitles
.filter(subtitle => isExtension(subtitle.title, allowedExtensions))
.sort((a, b) => b.size - a.size)
.map(subtitle => ({
infoHash: subtitle.infoHash,
fileIndex: subtitle.fileIndex,
title: subtitle.title,
lang: parseLanguage(subtitle.title, record),
}));
}
function parseLanguage(title, record) {
const subtitlePathParts = title.split('/');
const subtitleFileName = subtitlePathParts.pop();
const videoFileName = record.title.split('/').pop().replace(/\.\w{2,5}$/, '');
const fileNameLanguage = getSingleLanguage(subtitleFileName.replace(videoFileName, ''));
if (fileNameLanguage) {
return fileNameLanguage;
}
const folderName = subtitlePathParts.join('/');
const folderNameLanguage = getSingleLanguage(folderName.replace(videoFileName, ''));
if (folderNameLanguage) {
return folderNameLanguage
}
return getFileNameLanguageCode(subtitleFileName) || 'Unknown';
}
function getSingleLanguage(title) {
const parsedInfo = parse(title);
const languages = (parsedInfo.languages || []).filter(language => !ignoreSet.has(language));
return languages.length === 1 ? languageMapping[languages[0]] : undefined;
}
function getFileNameLanguageCode(fileName) {
const match = fileName.match(/(?:(?:^|[._ ])([A-Za-z][a-z]{1,2})|\[([a-z]{2,3})])\.\w{3,4}$/);
return match?.[1]?.toLowerCase();
}

View File

@@ -0,0 +1,26 @@
export function extractSize(title) {
const seedersMatch = title.match(/💾 ([\d.]+ \w+)/);
return seedersMatch && parseSize(seedersMatch[1]) || 0;
}
export function extractProvider(title) {
const match = title.match(/⚙.* ([^ \n]+)/);
return match?.[1]?.toLowerCase();
}
export function parseSize(sizeText) {
if (!sizeText) {
return 0;
}
let scale = 1;
if (sizeText.includes('TB')) {
scale = 1024 * 1024 * 1024 * 1024
} else if (sizeText.includes('GB')) {
scale = 1024 * 1024 * 1024
} else if (sizeText.includes('MB')) {
scale = 1024 * 1024;
} else if (sizeText.includes('kB')) {
scale = 1024;
}
return Math.floor(parseFloat(sizeText.replace(/,/g, '')) * scale);
}

View File

@@ -0,0 +1,6 @@
export const Type = {
MOVIE: 'movie',
SERIES: 'series',
ANIME: 'anime',
OTHER: 'other'
};

View File

@@ -0,0 +1,195 @@
import AllDebridClient from 'all-debrid-api';
import { isVideo, isArchive } from '../lib/extension.js';
import { getMagnetLink } from '../lib/magnetHelper.js';
import { Type } from '../lib/types.js';
import { BadTokenError, AccessDeniedError, sameFilename } from './mochHelper.js';
import StaticResponse from './static.js';
const KEY = 'alldebrid';
const AGENT = 'knightcrawler';
export async function getCachedStreams(streams, apiKey) {
const options = await getDefaultOptions();
const AD = new AllDebridClient(apiKey, options);
const hashes = streams.map(stream => stream.infoHash);
const available = await AD.magnet.instant(hashes)
.catch(error => {
if (toCommonError(error)) {
return Promise.reject(error);
}
console.warn(`Failed AllDebrid cached [${hashes[0]}] torrent availability request:`, error);
return undefined;
});
return available?.data?.magnets && streams
.reduce((mochStreams, stream) => {
const cachedEntry = available.data.magnets.find(magnet => stream.infoHash === magnet.hash.toLowerCase());
const streamTitleParts = stream.title.replace(/\n👤.*/s, '').split('\n');
const fileName = streamTitleParts[streamTitleParts.length - 1];
const fileIndex = streamTitleParts.length === 2 ? stream.fileIdx : null;
const encodedFileName = encodeURIComponent(fileName);
mochStreams[stream.infoHash] = {
url: `${apiKey}/${stream.infoHash}/${encodedFileName}/${fileIndex}`,
cached: cachedEntry?.instant
}
return mochStreams;
}, {})
}
export async function getCatalog(apiKey, offset = 0) {
if (offset > 0) {
return [];
}
const options = await getDefaultOptions();
const AD = new AllDebridClient(apiKey, options);
return AD.magnet.status()
.then(response => response.data.magnets)
.then(torrents => (torrents || [])
.filter(torrent => torrent && statusReady(torrent.statusCode))
.map(torrent => ({
id: `${KEY}:${torrent.id}`,
type: Type.OTHER,
name: torrent.filename
})));
}
export async function getItemMeta(itemId, apiKey) {
const options = await getDefaultOptions();
const AD = new AllDebridClient(apiKey, options);
return AD.magnet.status(itemId)
.then(response => response.data.magnets)
.then(torrent => ({
id: `${KEY}:${torrent.id}`,
type: Type.OTHER,
name: torrent.filename,
infoHash: torrent.hash.toLowerCase(),
videos: torrent.links
.filter(file => isVideo(file.filename))
.map((file, index) => ({
id: `${KEY}:${torrent.id}:${index}`,
title: file.filename,
released: new Date(torrent.uploadDate * 1000 - index).toISOString(),
streams: [{ url: `${apiKey}/${torrent.hash.toLowerCase()}/${encodeURIComponent(file.filename)}/${index}` }]
}))
}))
}
export async function resolve({ ip, apiKey, infoHash, cachedEntryInfo, fileIndex }) {
console.log(`Unrestricting AllDebrid ${infoHash} [${fileIndex}]`);
const options = await getDefaultOptions(ip);
const AD = new AllDebridClient(apiKey, options);
return _resolve(AD, infoHash, cachedEntryInfo, fileIndex)
.catch(error => {
if (errorExpiredSubscriptionError(error)) {
console.log(`Access denied to AllDebrid ${infoHash} [${fileIndex}]`);
return StaticResponse.FAILED_ACCESS;
} else if (error.code === 'MAGNET_TOO_MANY') {
console.log(`Deleting and retrying adding to AllDebrid ${infoHash} [${fileIndex}]...`);
return _deleteAndRetry(AD, infoHash, cachedEntryInfo, fileIndex);
}
return Promise.reject(`Failed AllDebrid adding torrent ${JSON.stringify(error)}`);
});
}
async function _resolve(AD, infoHash, cachedEntryInfo, fileIndex) {
const torrent = await _createOrFindTorrent(AD, infoHash);
if (torrent && statusReady(torrent.statusCode)) {
return _unrestrictLink(AD, torrent, cachedEntryInfo, fileIndex);
} else if (torrent && statusDownloading(torrent.statusCode)) {
console.log(`Downloading to AllDebrid ${infoHash} [${fileIndex}]...`);
return StaticResponse.DOWNLOADING;
} else if (torrent && statusHandledError(torrent.statusCode)) {
console.log(`Retrying downloading to AllDebrid ${infoHash} [${fileIndex}]...`);
return _retryCreateTorrent(AD, infoHash, cachedEntryInfo, fileIndex);
}
return Promise.reject(`Failed AllDebrid adding torrent ${JSON.stringify(torrent)}`);
}
async function _createOrFindTorrent(AD, infoHash) {
return _findTorrent(AD, infoHash)
.catch(() => _createTorrent(AD, infoHash));
}
async function _retryCreateTorrent(AD, infoHash, encodedFileName, fileIndex) {
const newTorrent = await _createTorrent(AD, infoHash);
return newTorrent && statusReady(newTorrent.statusCode)
? _unrestrictLink(AD, newTorrent, encodedFileName, fileIndex)
: StaticResponse.FAILED_DOWNLOAD;
}
async function _deleteAndRetry(AD, infoHash, encodedFileName, fileIndex) {
const torrents = await AD.magnet.status().then(response => response.data.magnets);
const lastTorrent = torrents[torrents.length - 1];
return AD.magnet.delete(lastTorrent.id)
.then(() => _retryCreateTorrent(AD, infoHash, encodedFileName, fileIndex));
}
async function _findTorrent(AD, infoHash) {
const torrents = await AD.magnet.status().then(response => response.data.magnets);
const foundTorrents = torrents.filter(torrent => torrent.hash.toLowerCase() === infoHash);
const nonFailedTorrent = foundTorrents.find(torrent => !statusError(torrent.statusCode));
const foundTorrent = nonFailedTorrent || foundTorrents[0];
return foundTorrent || Promise.reject('No recent torrent found');
}
async function _createTorrent(AD, infoHash) {
const magnetLink = await getMagnetLink(infoHash);
const uploadResponse = await AD.magnet.upload(magnetLink);
const torrentId = uploadResponse.data.magnets[0].id;
return AD.magnet.status(torrentId).then(statusResponse => statusResponse.data.magnets);
}
async function _unrestrictLink(AD, torrent, encodedFileName, fileIndex) {
const targetFileName = decodeURIComponent(encodedFileName);
const videos = torrent.links.filter(link => isVideo(link.filename));
const targetVideo = Number.isInteger(fileIndex)
? videos.find(video => sameFilename(targetFileName, video.filename))
: videos.sort((a, b) => b.size - a.size)[0];
if (!targetVideo && torrent.links.every(link => isArchive(link.filename))) {
console.log(`Only AllDebrid archive is available for [${torrent.hash}] ${encodedFileName}`)
return StaticResponse.FAILED_RAR;
}
if (!targetVideo || !targetVideo.link || !targetVideo.link.length) {
return Promise.reject(`No AllDebrid links found for [${torrent.hash}] ${encodedFileName}`);
}
const unrestrictedLink = await AD.link.unlock(targetVideo.link).then(response => response.data.link);
console.log(`Unrestricted AllDebrid ${torrent.hash} [${fileIndex}] to ${unrestrictedLink}`);
return unrestrictedLink;
}
async function getDefaultOptions(ip) {
return { base_agent: AGENT, timeout: 10000 };
}
export function toCommonError(error) {
if (error && error.code === 'AUTH_BAD_APIKEY') {
return BadTokenError;
}
if (error && error.code === 'AUTH_USER_BANNED') {
return AccessDeniedError;
}
return undefined;
}
function statusError(statusCode) {
return [5, 6, 7, 8, 9, 10, 11].includes(statusCode);
}
function statusHandledError(statusCode) {
return [5, 7, 9, 10, 11].includes(statusCode);
}
function statusDownloading(statusCode) {
return [0, 1, 2, 3].includes(statusCode);
}
function statusReady(statusCode) {
return statusCode === 4;
}
function errorExpiredSubscriptionError(error) {
return ['AUTH_BAD_APIKEY', 'MUST_BE_PREMIUM', 'MAGNET_MUST_BE_PREMIUM', 'FREE_TRIAL_LIMIT_REACHED', 'AUTH_USER_BANNED']
.includes(error.code);
}

View File

@@ -0,0 +1,155 @@
import DebridLinkClient from 'debrid-link-api';
import { isVideo, isArchive } from '../lib/extension.js';
import { getMagnetLink } from '../lib/magnetHelper.js';
import { Type } from '../lib/types.js';
import { chunkArray, BadTokenError } from './mochHelper.js';
import StaticResponse from './static.js';
const KEY = 'debridlink';
export async function getCachedStreams(streams, apiKey) {
const options = await getDefaultOptions();
const DL = new DebridLinkClient(apiKey, options);
const hashBatches = chunkArray(streams.map(stream => stream.infoHash), 50)
.map(batch => batch.join(','));
const available = await Promise.all(hashBatches.map(hashes => DL.seedbox.cached(hashes)))
.then(results => results.map(result => result.value))
.then(results => results.reduce((all, result) => Object.assign(all, result), {}))
.catch(error => {
if (toCommonError(error)) {
return Promise.reject(error);
}
console.warn('Failed DebridLink cached torrent availability request:', error);
return undefined;
});
return available && streams
.reduce((mochStreams, stream) => {
const cachedEntry = available[stream.infoHash];
mochStreams[stream.infoHash] = {
url: `${apiKey}/${stream.infoHash}/null/${stream.fileIdx}`,
cached: !!cachedEntry
};
return mochStreams;
}, {})
}
export async function getCatalog(apiKey, offset = 0) {
if (offset > 0) {
return [];
}
const options = await getDefaultOptions();
const DL = new DebridLinkClient(apiKey, options);
return DL.seedbox.list()
.then(response => response.value)
.then(torrents => (torrents || [])
.filter(torrent => torrent && statusReady(torrent))
.map(torrent => ({
id: `${KEY}:${torrent.id}`,
type: Type.OTHER,
name: torrent.name
})));
}
export async function getItemMeta(itemId, apiKey, ip) {
const options = await getDefaultOptions(ip);
const DL = new DebridLinkClient(apiKey, options);
return DL.seedbox.list(itemId)
.then(response => response.value[0])
.then(torrent => ({
id: `${KEY}:${torrent.id}`,
type: Type.OTHER,
name: torrent.name,
infoHash: torrent.hashString.toLowerCase(),
videos: torrent.files
.filter(file => isVideo(file.name))
.map((file, index) => ({
id: `${KEY}:${torrent.id}:${index}`,
title: file.name,
released: new Date(torrent.created * 1000 - index).toISOString(),
streams: [{ url: file.downloadUrl }]
}))
}))
}
export async function resolve({ ip, apiKey, infoHash, fileIndex }) {
console.log(`Unrestricting DebridLink ${infoHash} [${fileIndex}]`);
const options = await getDefaultOptions(ip);
const DL = new DebridLinkClient(apiKey, options);
return _resolve(DL, infoHash, fileIndex)
.catch(error => {
if (errorExpiredSubscriptionError(error)) {
console.log(`Access denied to DebridLink ${infoHash} [${fileIndex}]`);
return StaticResponse.FAILED_ACCESS;
}
return Promise.reject(`Failed DebridLink adding torrent ${JSON.stringify(error)}`);
});
}
async function _resolve(DL, infoHash, fileIndex) {
const torrent = await _createOrFindTorrent(DL, infoHash);
if (torrent && statusReady(torrent)) {
return _unrestrictLink(DL, torrent, fileIndex);
} else if (torrent && statusDownloading(torrent)) {
console.log(`Downloading to DebridLink ${infoHash} [${fileIndex}]...`);
return StaticResponse.DOWNLOADING;
}
return Promise.reject(`Failed DebridLink adding torrent ${JSON.stringify(torrent)}`);
}
async function _createOrFindTorrent(DL, infoHash) {
return _findTorrent(DL, infoHash)
.catch(() => _createTorrent(DL, infoHash));
}
async function _findTorrent(DL, infoHash) {
const torrents = await DL.seedbox.list().then(response => response.value);
const foundTorrents = torrents.filter(torrent => torrent.hashString.toLowerCase() === infoHash);
return foundTorrents[0] || Promise.reject('No recent torrent found');
}
async function _createTorrent(DL, infoHash) {
const magnetLink = await getMagnetLink(infoHash);
const uploadResponse = await DL.seedbox.add(magnetLink, null, true);
return uploadResponse.value;
}
async function _unrestrictLink(DL, torrent, fileIndex) {
const targetFile = Number.isInteger(fileIndex)
? torrent.files[fileIndex]
: torrent.files.filter(file => file.downloadPercent === 100).sort((a, b) => b.size - a.size)[0];
if (!targetFile && torrent.files.every(file => isArchive(file.downloadUrl))) {
console.log(`Only DebridLink archive is available for [${torrent.hashString}] ${fileIndex}`)
return StaticResponse.FAILED_RAR;
}
if (!targetFile || !targetFile.downloadUrl) {
return Promise.reject(`No DebridLink links found for index ${fileIndex} in: ${JSON.stringify(torrent)}`);
}
console.log(`Unrestricted DebridLink ${torrent.hashString} [${fileIndex}] to ${targetFile.downloadUrl}`);
return targetFile.downloadUrl;
}
async function getDefaultOptions(ip) {
return { ip, timeout: 10000 };
}
export function toCommonError(error) {
if (error === 'badToken') {
return BadTokenError;
}
return undefined;
}
function statusDownloading(torrent) {
return torrent.downloadPercent < 100
}
function statusReady(torrent) {
return torrent.downloadPercent === 100;
}
function errorExpiredSubscriptionError(error) {
return ['freeServerOverload', 'maxTorrent', 'maxLink', 'maxLinkHost', 'maxData', 'maxDataHost'].includes(error);
}

240
src/addon/src/moch/moch.js Normal file
View File

@@ -0,0 +1,240 @@
import namedQueue from 'named-queue';
import { cacheWrapResolvedUrl } from '../lib/cache.js';
import { timeout } from '../lib/promises.js';
import * as alldebrid from './alldebrid.js';
import * as debridlink from './debridlink.js';
import { BadTokenError, streamFilename, AccessDeniedError, enrichMeta } from './mochHelper.js';
import * as offcloud from './offcloud.js';
import * as options from './options.js';
import * as premiumize from './premiumize.js';
import * as putio from './putio.js';
import * as realdebrid from './realdebrid.js';
import StaticResponse, { isStaticUrl } from './static.js';
const RESOLVE_TIMEOUT = 2 * 60 * 1000; // 2 minutes
const MIN_API_KEY_SYMBOLS = 15;
const TOKEN_BLACKLIST = [];
export const MochOptions = {
realdebrid: {
key: 'realdebrid',
instance: realdebrid,
name: "RealDebrid",
shortName: 'RD',
catalog: true
},
premiumize: {
key: 'premiumize',
instance: premiumize,
name: 'Premiumize',
shortName: 'PM',
catalog: true
},
alldebrid: {
key: 'alldebrid',
instance: alldebrid,
name: 'AllDebrid',
shortName: 'AD',
catalog: true
},
debridlink: {
key: 'debridlink',
instance: debridlink,
name: 'DebridLink',
shortName: 'DL',
catalog: true
},
offcloud: {
key: 'offcloud',
instance: offcloud,
name: 'Offcloud',
shortName: 'OC',
catalog: true
},
putio: {
key: 'putio',
instance: putio,
name: 'Put.io',
shortName: 'Putio',
catalog: true
}
};
const unrestrictQueues = {}
Object.values(MochOptions)
.map(moch => moch.key)
.forEach(mochKey => unrestrictQueues[mochKey] = new namedQueue((task, callback) => task.method()
.then(result => callback(false, result))
.catch((error => callback(error))), 200));
export function hasMochConfigured(config) {
return Object.keys(MochOptions).find(moch => config?.[moch])
}
export async function applyMochs(streams, config) {
if (!streams?.length || !hasMochConfigured(config)) {
return streams;
}
return Promise.all(Object.keys(config)
.filter(configKey => MochOptions[configKey])
.map(configKey => MochOptions[configKey])
.map(moch => {
if (isInvalidToken(config[moch.key], moch.key)) {
return { moch, error: BadTokenError };
}
return moch.instance.getCachedStreams(streams, config[moch.key])
.then(mochStreams => ({ moch, mochStreams }))
.catch(rawError => {
const error = moch.instance.toCommonError(rawError) || rawError;
if (error === BadTokenError) {
blackListToken(config[moch.key], moch.key);
}
return { moch, error };
})
}))
.then(results => processMochResults(streams, config, results));
}
export async function resolve(parameters) {
const moch = MochOptions[parameters.mochKey];
if (!moch) {
return Promise.reject(new Error(`Not a valid moch provider: ${parameters.mochKey}`));
}
if (!parameters.apiKey || !parameters.infoHash || !parameters.cachedEntryInfo) {
return Promise.reject(new Error("No valid parameters passed"));
}
const id = `${parameters.ip}_${parameters.mochKey}_${parameters.apiKey}_${parameters.infoHash}_${parameters.fileIndex}`;
const method = () => timeout(RESOLVE_TIMEOUT, cacheWrapResolvedUrl(id, () => moch.instance.resolve(parameters)))
.catch(error => {
console.warn(error);
return StaticResponse.FAILED_UNEXPECTED;
})
.then(url => isStaticUrl(url) ? `${parameters.host}/${url}` : url);
const unrestrictQueue = unrestrictQueues[moch.key];
return new Promise(((resolve, reject) => {
unrestrictQueue.push({ id, method }, (error, result) => result ? resolve(result) : reject(error));
}));
}
export async function getMochCatalog(mochKey, config) {
const moch = MochOptions[mochKey];
if (!moch) {
return Promise.reject(new Error(`Not a valid moch provider: ${mochKey}`));
}
if (isInvalidToken(config[mochKey], mochKey)) {
return Promise.reject(new Error(`Invalid API key for moch provider: ${mochKey}`));
}
return moch.instance.getCatalog(config[moch.key], config.skip, config.ip)
.catch(rawError => {
const commonError = moch.instance.toCommonError(rawError);
if (commonError === BadTokenError) {
blackListToken(config[moch.key], moch.key);
}
return commonError ? [] : Promise.reject(rawError);
});
}
export async function getMochItemMeta(mochKey, itemId, config) {
const moch = MochOptions[mochKey];
if (!moch) {
return Promise.reject(new Error(`Not a valid moch provider: ${mochKey}`));
}
return moch.instance.getItemMeta(itemId, config[moch.key], config.ip)
.then(meta => enrichMeta(meta))
.then(meta => {
meta.videos.forEach(video => video.streams.forEach(stream => {
if (!stream.url.startsWith('http')) {
stream.url = `${config.host}/${moch.key}/${stream.url}/${streamFilename(video)}`
}
stream.behaviorHints = { bingeGroup: itemId }
}))
return meta;
});
}
function processMochResults(streams, config, results) {
const errorResults = results
.map(result => errorStreamResponse(result.moch.key, result.error, config))
.filter(errorResponse => errorResponse);
if (errorResults.length) {
return errorResults;
}
const includeTorrentLinks = options.includeTorrentLinks(config);
const excludeDownloadLinks = options.excludeDownloadLinks(config);
const mochResults = results.filter(result => result?.mochStreams);
const cachedStreams = mochResults
.reduce((resultStreams, mochResult) => populateCachedLinks(resultStreams, mochResult, config), streams);
const resultStreams = excludeDownloadLinks ? cachedStreams : populateDownloadLinks(cachedStreams, mochResults, config);
return includeTorrentLinks ? resultStreams : resultStreams.filter(stream => stream.url);
}
function populateCachedLinks(streams, mochResult, config) {
return streams.map(stream => {
const cachedEntry = stream.infoHash && mochResult.mochStreams[stream.infoHash];
if (cachedEntry?.cached) {
return {
name: `[${mochResult.moch.shortName}+] ${stream.name}`,
title: stream.title,
url: `${config.host}/${mochResult.moch.key}/${cachedEntry.url}/${streamFilename(stream)}`,
behaviorHints: stream.behaviorHints
};
}
return stream;
});
}
function populateDownloadLinks(streams, mochResults, config) {
const torrentStreams = streams.filter(stream => stream.infoHash);
const seededStreams = streams.filter(stream => !stream.title.includes('👤 0'));
torrentStreams.forEach(stream => mochResults.forEach(mochResult => {
const cachedEntry = mochResult.mochStreams[stream.infoHash];
const isCached = cachedEntry?.cached;
if (!isCached && isHealthyStreamForDebrid(seededStreams, stream)) {
streams.push({
name: `[${mochResult.moch.shortName} download] ${stream.name}`,
title: stream.title,
url: `${config.host}/${mochResult.moch.key}/${cachedEntry.url}/${streamFilename(stream)}`,
behaviorHints: stream.behaviorHints
})
}
}));
return streams;
}
function isHealthyStreamForDebrid(streams, stream) {
const isZeroSeeders = stream.title.includes('👤 0');
const is4kStream = stream.name.includes('4k');
const isNotEnoughOptions = streams.length <= 5;
return !isZeroSeeders || is4kStream || isNotEnoughOptions;
}
function isInvalidToken(token, mochKey) {
return token.length < MIN_API_KEY_SYMBOLS || TOKEN_BLACKLIST.includes(`${mochKey}|${token}`);
}
function blackListToken(token, mochKey) {
const tokenKey = `${mochKey}|${token}`;
console.log(`Blacklisting invalid token: ${tokenKey}`)
TOKEN_BLACKLIST.push(tokenKey);
}
function errorStreamResponse(mochKey, error, config) {
if (error === BadTokenError) {
return {
name: `KnightCrawler\n${MochOptions[mochKey].shortName} error`,
title: `Invalid ${MochOptions[mochKey].name} ApiKey/Token!`,
url: `${config.host}/${StaticResponse.FAILED_ACCESS}`
};
}
if (error === AccessDeniedError) {
return {
name: `KnightCrawler\n${MochOptions[mochKey].shortName} error`,
title: `Expired/invalid ${MochOptions[mochKey].name} subscription!`,
url: `${config.host}/${StaticResponse.FAILED_ACCESS}`
};
}
return undefined;
}

View File

@@ -0,0 +1,63 @@
import * as repository from '../lib/repository.js';
const METAHUB_URL = 'https://images.metahub.space'
export const BadTokenError = { code: 'BAD_TOKEN' }
export const AccessDeniedError = { code: 'ACCESS_DENIED' }
export function chunkArray(arr, size) {
return arr.length > size
? [arr.slice(0, size), ...chunkArray(arr.slice(size), size)]
: [arr];
}
export function streamFilename(stream) {
const titleParts = stream.title.replace(/\n👤.*/s, '').split('\n');
const filename = titleParts.pop().split('/').pop();
return encodeURIComponent(filename)
}
export async function enrichMeta(itemMeta) {
const infoHashes = [...new Set([itemMeta.infoHash]
.concat(itemMeta.videos.map(video => video.infoHash))
.filter(infoHash => infoHash))];
const files = infoHashes.length ? await repository.getFiles(infoHashes).catch(() => []) : [];
const commonImdbId = itemMeta.infoHash && mostCommonValue(files.map(file => file.imdbId));
if (files.length) {
return {
...itemMeta,
logo: commonImdbId && `${METAHUB_URL}/logo/medium/${commonImdbId}/img`,
poster: commonImdbId && `${METAHUB_URL}/poster/medium/${commonImdbId}/img`,
background: commonImdbId && `${METAHUB_URL}/background/medium/${commonImdbId}/img`,
videos: itemMeta.videos.map(video => {
const file = files.find(file => sameFilename(video.title, file.title));
if (file?.imdbId) {
if (file.imdbSeason && file.imdbEpisode) {
video.id = `${file.imdbId}:${file.imdbSeason}:${file.imdbEpisode}`;
video.season = file.imdbSeason;
video.episode = file.imdbEpisode;
video.thumbnail = `https://episodes.metahub.space/${file.imdbId}/${video.season}/${video.episode}/w780.jpg`
} else {
video.id = file.imdbId;
video.thumbnail = `${METAHUB_URL}/background/small/${file.imdbId}/img`;
}
}
return video;
})
}
}
return itemMeta
}
export function sameFilename(filename, expectedFilename) {
const offset = filename.length - expectedFilename.length;
for (let i = 0; i < expectedFilename.length; i++) {
if (filename[offset + i] !== expectedFilename[i] && expectedFilename[i] !== '<27>') {
return false;
}
}
return true;
}
function mostCommonValue(array) {
return array.sort((a, b) => array.filter(v => v === a).length - array.filter(v => v === b).length).pop();
}

View File

@@ -0,0 +1,186 @@
import magnet from 'magnet-uri';
import OffcloudClient from 'offcloud-api';
import { isVideo } from '../lib/extension.js';
import { getMagnetLink } from '../lib/magnetHelper.js';
import { Type } from '../lib/types.js';
import { chunkArray, BadTokenError, sameFilename } from './mochHelper.js';
import StaticResponse from './static.js';
const KEY = 'offcloud';
export async function getCachedStreams(streams, apiKey) {
const options = await getDefaultOptions();
const OC = new OffcloudClient(apiKey, options);
const hashBatches = chunkArray(streams.map(stream => stream.infoHash), 100);
const available = await Promise.all(hashBatches.map(hashes => OC.instant.cache(hashes)))
.then(results => results.map(result => result.cachedItems))
.then(results => results.reduce((all, result) => all.concat(result), []))
.catch(error => {
if (toCommonError(error)) {
return Promise.reject(error);
}
console.warn('Failed Offcloud cached torrent availability request:', error);
return undefined;
});
return available && streams
.reduce((mochStreams, stream) => {
const isCached = available.includes(stream.infoHash);
const streamTitleParts = stream.title.replace(/\n👤.*/s, '').split('\n');
const fileName = streamTitleParts[streamTitleParts.length - 1];
const fileIndex = streamTitleParts.length === 2 ? stream.fileIdx : null;
const encodedFileName = encodeURIComponent(fileName);
mochStreams[stream.infoHash] = {
url: `${apiKey}/${stream.infoHash}/${encodedFileName}/${fileIndex}`,
cached: isCached
};
return mochStreams;
}, {})
}
export async function getCatalog(apiKey, offset = 0) {
if (offset > 0) {
return [];
}
const options = await getDefaultOptions();
const OC = new OffcloudClient(apiKey, options);
return OC.cloud.history()
.then(torrents => torrents)
.then(torrents => (torrents || [])
.map(torrent => ({
id: `${KEY}:${torrent.requestId}`,
type: Type.OTHER,
name: torrent.fileName
})));
}
export async function getItemMeta(itemId, apiKey, ip) {
const options = await getDefaultOptions(ip);
const OC = new OffcloudClient(apiKey, options);
const torrents = await OC.cloud.history();
const torrent = torrents.find(torrent => torrent.requestId === itemId)
const infoHash = torrent && magnet.decode(torrent.originalLink).infoHash
const createDate = torrent ? new Date(torrent.createdOn) : new Date();
return _getFileUrls(OC, torrent)
.then(files => ({
id: `${KEY}:${itemId}`,
type: Type.OTHER,
name: torrent.name,
infoHash: infoHash,
videos: files
.filter(file => isVideo(file))
.map((file, index) => ({
id: `${KEY}:${itemId}:${index}`,
title: file.split('/').pop(),
released: new Date(createDate.getTime() - index).toISOString(),
streams: [{ url: file }]
}))
}))
}
export async function resolve({ ip, apiKey, infoHash, cachedEntryInfo, fileIndex }) {
console.log(`Unrestricting Offcloud ${infoHash} [${fileIndex}]`);
const options = await getDefaultOptions(ip);
const OC = new OffcloudClient(apiKey, options);
return _resolve(OC, infoHash, cachedEntryInfo, fileIndex)
.catch(error => {
if (errorExpiredSubscriptionError(error)) {
console.log(`Access denied to Offcloud ${infoHash} [${fileIndex}]`);
return StaticResponse.FAILED_ACCESS;
}
return Promise.reject(`Failed Offcloud adding torrent ${JSON.stringify(error)}`);
});
}
async function _resolve(OC, infoHash, cachedEntryInfo, fileIndex) {
const torrent = await _createOrFindTorrent(OC, infoHash)
.then(info => info.requestId ? OC.cloud.status(info.requestId) : Promise.resolve(info))
.then(info => info.status || info);
if (torrent && statusReady(torrent)) {
return _unrestrictLink(OC, infoHash, torrent, cachedEntryInfo, fileIndex);
} else if (torrent && statusDownloading(torrent)) {
console.log(`Downloading to Offcloud ${infoHash} [${fileIndex}]...`);
return StaticResponse.DOWNLOADING;
} else if (torrent && statusError(torrent)) {
console.log(`Retry failed download in Offcloud ${infoHash} [${fileIndex}]...`);
return _retryCreateTorrent(OC, infoHash, cachedEntryInfo, fileIndex);
}
return Promise.reject(`Failed Offcloud adding torrent ${JSON.stringify(torrent)}`);
}
async function _createOrFindTorrent(OC, infoHash) {
return _findTorrent(OC, infoHash)
.catch(() => _createTorrent(OC, infoHash));
}
async function _findTorrent(OC, infoHash) {
const torrents = await OC.cloud.history();
const foundTorrents = torrents.filter(torrent => torrent.originalLink.toLowerCase().includes(infoHash));
const nonFailedTorrent = foundTorrents.find(torrent => !statusError(torrent));
const foundTorrent = nonFailedTorrent || foundTorrents[0];
return foundTorrent || Promise.reject('No recent torrent found');
}
async function _createTorrent(OC, infoHash) {
const magnetLink = await getMagnetLink(infoHash);
return OC.cloud.download(magnetLink)
}
async function _retryCreateTorrent(OC, infoHash, cachedEntryInfo, fileIndex) {
const newTorrent = await _createTorrent(OC, infoHash);
return newTorrent && statusReady(newTorrent.status)
? _unrestrictLink(OC, infoHash, newTorrent, cachedEntryInfo, fileIndex)
: StaticResponse.FAILED_DOWNLOAD;
}
async function _unrestrictLink(OC, infoHash, torrent, cachedEntryInfo, fileIndex) {
const targetFileName = decodeURIComponent(cachedEntryInfo);
const files = await _getFileUrls(OC, torrent)
const targetFile = files.find(file => sameFilename(targetFileName, file.split('/').pop()))
|| files.find(file => isVideo(file))
|| files.pop();
if (!targetFile) {
return Promise.reject(`No Offcloud links found for index ${fileIndex} in: ${JSON.stringify(torrent)}`);
}
console.log(`Unrestricted Offcloud ${infoHash} [${fileIndex}] to ${targetFile}`);
return targetFile;
}
async function _getFileUrls(OC, torrent) {
return OC.cloud.explore(torrent.requestId)
.catch(error => {
if (error === 'Bad archive') {
return [`https://${torrent.server}.offcloud.com/cloud/download/${torrent.requestId}/${torrent.fileName}`];
}
throw error;
})
}
async function getDefaultOptions(ip) {
return { ip, timeout: 10000 };
}
export function toCommonError(error) {
if (error?.error === 'NOAUTH' || error?.message?.startsWith('Cannot read property')) {
return BadTokenError;
}
return undefined;
}
function statusDownloading(torrent) {
return ['downloading', 'created'].includes(torrent.status);
}
function statusError(torrent) {
return ['error', 'canceled'].includes(torrent.status);
}
function statusReady(torrent) {
return torrent.status === 'downloaded';
}
function errorExpiredSubscriptionError(error) {
return error?.includes && (error.includes('not_available') || error.includes('NOAUTH') || error.includes('premium membership'));
}

View File

@@ -0,0 +1,29 @@
export const DebridOptions = {
key: 'debridoptions',
options: {
noDownloadLinks: {
key: 'nodownloadlinks',
description: 'Don\'t show download to debrid links'
},
noCatalog: {
key: 'nocatalog',
description: 'Don\'t show debrid catalog'
},
torrentLinks: {
key: 'torrentlinks',
description: 'Show P2P torrent links for uncached'
}
}
}
export function excludeDownloadLinks(config) {
return config[DebridOptions.key]?.includes(DebridOptions.options.noDownloadLinks.key);
}
export function includeTorrentLinks(config) {
return config[DebridOptions.key]?.includes(DebridOptions.options.torrentLinks.key);
}
export function showDebridCatalog(config) {
return !config[DebridOptions.key]?.includes(DebridOptions.options.noCatalog.key);
}

View File

@@ -0,0 +1,195 @@
import magnet from 'magnet-uri';
import PremiumizeClient from 'premiumize-api';
import { isVideo, isArchive } from '../lib/extension.js';
import { getMagnetLink } from '../lib/magnetHelper.js';
import { Type } from '../lib/types.js';
import { BadTokenError, chunkArray, sameFilename } from './mochHelper.js';
import StaticResponse from './static.js';
const KEY = 'premiumize';
export async function getCachedStreams(streams, apiKey) {
const options = await getDefaultOptions();
const PM = new PremiumizeClient(apiKey, options);
return Promise.all(chunkArray(streams, 100)
.map(chunkedStreams => _getCachedStreams(PM, apiKey, chunkedStreams)))
.then(results => results.reduce((all, result) => Object.assign(all, result), {}));
}
async function _getCachedStreams(PM, apiKey, streams) {
const hashes = streams.map(stream => stream.infoHash);
return PM.cache.check(hashes)
.catch(error => {
if (toCommonError(error)) {
return Promise.reject(error);
}
console.warn('Failed Premiumize cached torrent availability request:', error);
return undefined;
})
.then(available => streams
.reduce((mochStreams, stream, index) => {
const streamTitleParts = stream.title.replace(/\n👤.*/s, '').split('\n');
const fileName = streamTitleParts[streamTitleParts.length - 1];
const fileIndex = streamTitleParts.length === 2 ? stream.fileIdx : null;
const encodedFileName = encodeURIComponent(fileName);
mochStreams[`${stream.infoHash}@${stream.fileIdx}`] = {
url: `${apiKey}/${stream.infoHash}/${encodedFileName}/${fileIndex}`,
cached: available?.response[index]
};
return mochStreams;
}, {}));
}
export async function getCatalog(apiKey, offset = 0) {
if (offset > 0) {
return [];
}
const options = await getDefaultOptions();
const PM = new PremiumizeClient(apiKey, options);
return PM.folder.list()
.then(response => response.content)
.then(torrents => (torrents || [])
.filter(torrent => torrent && torrent.type === 'folder')
.map(torrent => ({
id: `${KEY}:${torrent.id}`,
type: Type.OTHER,
name: torrent.name
})));
}
export async function getItemMeta(itemId, apiKey, ip) {
const options = await getDefaultOptions();
const PM = new PremiumizeClient(apiKey, options);
const rootFolder = await PM.folder.list(itemId, null);
const infoHash = await _findInfoHash(PM, itemId);
return getFolderContents(PM, itemId, ip)
.then(contents => ({
id: `${KEY}:${itemId}`,
type: Type.OTHER,
name: rootFolder.name,
infoHash: infoHash,
videos: contents
.map((file, index) => ({
id: `${KEY}:${file.id}:${index}`,
title: file.name,
released: new Date(file.created_at * 1000 - index).toISOString(),
streams: [{ url: file.link || file.stream_link }]
}))
}))
}
async function getFolderContents(PM, itemId, ip, folderPrefix = '') {
return PM.folder.list(itemId, null, ip)
.then(response => response.content)
.then(contents => Promise.all(contents
.filter(content => content.type === 'folder')
.map(content => getFolderContents(PM, content.id, ip, [folderPrefix, content.name].join('/'))))
.then(otherContents => otherContents.reduce((a, b) => a.concat(b), []))
.then(otherContents => contents
.filter(content => content.type === 'file' && isVideo(content.name))
.map(content => ({ ...content, name: [folderPrefix, content.name].join('/') }))
.concat(otherContents)));
}
export async function resolve({ ip, isBrowser, apiKey, infoHash, cachedEntryInfo, fileIndex }) {
console.log(`Unrestricting Premiumize ${infoHash} [${fileIndex}] for IP ${ip} from browser=${isBrowser}`);
const options = await getDefaultOptions();
const PM = new PremiumizeClient(apiKey, options);
return _getCachedLink(PM, infoHash, cachedEntryInfo, fileIndex, ip, isBrowser)
.catch(() => _resolve(PM, infoHash, cachedEntryInfo, fileIndex, ip, isBrowser))
.catch(error => {
if (error?.message?.includes('Account not premium.')) {
console.log(`Access denied to Premiumize ${infoHash} [${fileIndex}]`);
return StaticResponse.FAILED_ACCESS;
}
return Promise.reject(`Failed Premiumize adding torrent ${JSON.stringify(error)}`);
});
}
async function _resolve(PM, infoHash, cachedEntryInfo, fileIndex, ip, isBrowser) {
const torrent = await _createOrFindTorrent(PM, infoHash);
if (torrent && statusReady(torrent.status)) {
return _getCachedLink(PM, infoHash, cachedEntryInfo, fileIndex, ip, isBrowser);
} else if (torrent && statusDownloading(torrent.status)) {
console.log(`Downloading to Premiumize ${infoHash} [${fileIndex}]...`);
return StaticResponse.DOWNLOADING;
} else if (torrent && statusError(torrent.status)) {
console.log(`Retrying downloading to Premiumize ${infoHash} [${fileIndex}]...`);
return _retryCreateTorrent(PM, infoHash, cachedEntryInfo, fileIndex);
}
return Promise.reject(`Failed Premiumize adding torrent ${JSON.stringify(torrent)}`);
}
async function _getCachedLink(PM, infoHash, encodedFileName, fileIndex, ip, isBrowser) {
const cachedTorrent = await PM.transfer.directDownload(magnet.encode({ infoHash }), ip);
if (cachedTorrent?.content?.length) {
const targetFileName = decodeURIComponent(encodedFileName);
const videos = cachedTorrent.content.filter(file => isVideo(file.path));
const targetVideo = Number.isInteger(fileIndex)
? videos.find(video => sameFilename(video.path, targetFileName))
: videos.sort((a, b) => b.size - a.size)[0];
if (!targetVideo && videos.every(video => isArchive(video.path))) {
console.log(`Only Premiumize archive is available for [${infoHash}] ${fileIndex}`)
return StaticResponse.FAILED_RAR;
}
const streamLink = isBrowser && targetVideo.transcode_status === 'finished' && targetVideo.stream_link;
const unrestrictedLink = streamLink || targetVideo.link;
console.log(`Unrestricted Premiumize ${infoHash} [${fileIndex}] to ${unrestrictedLink}`);
return unrestrictedLink;
}
return Promise.reject('No cached entry found');
}
async function _createOrFindTorrent(PM, infoHash) {
return _findTorrent(PM, infoHash)
.catch(() => _createTorrent(PM, infoHash));
}
async function _findTorrent(PM, infoHash) {
const torrents = await PM.transfer.list().then(response => response.transfers);
const foundTorrents = torrents.filter(torrent => torrent.src.toLowerCase().includes(infoHash));
const nonFailedTorrent = foundTorrents.find(torrent => !statusError(torrent.statusCode));
const foundTorrent = nonFailedTorrent || foundTorrents[0];
return foundTorrent || Promise.reject('No recent torrent found');
}
async function _findInfoHash(PM, itemId) {
const torrents = await PM.transfer.list().then(response => response.transfers);
const foundTorrent = torrents.find(torrent => `${torrent.file_id}` === itemId || `${torrent.folder_id}` === itemId);
return foundTorrent?.src ? magnet.decode(foundTorrent.src).infoHash : undefined;
}
async function _createTorrent(PM, infoHash) {
const magnetLink = await getMagnetLink(infoHash);
return PM.transfer.create(magnetLink).then(() => _findTorrent(PM, infoHash));
}
async function _retryCreateTorrent(PM, infoHash, encodedFileName, fileIndex) {
const newTorrent = await _createTorrent(PM, infoHash).then(() => _findTorrent(PM, infoHash));
return newTorrent && statusReady(newTorrent.status)
? _getCachedLink(PM, infoHash, encodedFileName, fileIndex)
: StaticResponse.FAILED_DOWNLOAD;
}
export function toCommonError(error) {
if (error && error.message === 'Not logged in.') {
return BadTokenError;
}
return undefined;
}
function statusError(status) {
return ['deleted', 'error', 'timeout'].includes(status);
}
function statusDownloading(status) {
return ['waiting', 'queued', 'running'].includes(status);
}
function statusReady(status) {
return ['finished', 'seeding'].includes(status);
}
async function getDefaultOptions(ip) {
return { timeout: 5000 };
}

215
src/addon/src/moch/putio.js Normal file
View File

@@ -0,0 +1,215 @@
import PutioClient from '@putdotio/api-client'
import { decode } from "magnet-uri";
import { isVideo } from '../lib/extension.js';
import { getMagnetLink } from '../lib/magnetHelper.js';
import { delay } from '../lib/promises.js';
import { Type } from "../lib/types.js";
import { sameFilename } from "./mochHelper.js";
import StaticResponse from './static.js';
const PutioAPI = PutioClient.default;
const KEY = 'putio';
export async function getCachedStreams(streams, apiKey) {
return streams
.reduce((mochStreams, stream) => {
const streamTitleParts = stream.title.replace(/\n👤.*/s, '').split('\n');
const fileName = streamTitleParts[streamTitleParts.length - 1];
const fileIndex = streamTitleParts.length === 2 ? stream.fileIdx : null;
const encodedFileName = encodeURIComponent(fileName);
mochStreams[stream.infoHash] = {
url: `${apiKey}/${stream.infoHash}/${encodedFileName}/${fileIndex}`,
cached: false
};
return mochStreams;
}, {});
}
export async function getCatalog(apiKey, offset = 0) {
if (offset > 0) {
return [];
}
const Putio = createPutioAPI(apiKey)
return Putio.Files.Query(0)
.then(response => response?.body?.files)
.then(files => (files || [])
.map(file => ({
id: `${KEY}:${file.id}`,
type: Type.OTHER,
name: file.name
})));
}
export async function getItemMeta(itemId, apiKey) {
const Putio = createPutioAPI(apiKey)
const infoHash = await _findInfoHash(Putio, itemId)
return getFolderContents(Putio, itemId)
.then(contents => ({
id: `${KEY}:${itemId}`,
type: Type.OTHER,
name: contents.name,
infoHash: infoHash,
videos: contents
.map((file, index) => ({
id: `${KEY}:${file.id}:${index}`,
title: file.name,
released: new Date(file.created_at).toISOString(),
streams: [{ url: `${apiKey}/null/null/${file.id}` }]
}))
}))
}
async function getFolderContents(Putio, itemId, folderPrefix = '') {
return await Putio.Files.Query(itemId)
.then(response => response?.body)
.then(body => body?.files?.length ? body.files : [body?.parent].filter(x => x))
.then(contents => Promise.all(contents
.filter(content => content.file_type === 'FOLDER')
.map(content => getFolderContents(Putio, content.id, [folderPrefix, content.name].join('/'))))
.then(otherContents => otherContents.reduce((a, b) => a.concat(b), []))
.then(otherContents => contents
.filter(content => content.file_type === 'VIDEO')
.map(content => ({ ...content, name: [folderPrefix, content.name].join('/') }))
.concat(otherContents)));
}
export async function resolve({ ip, apiKey, infoHash, cachedEntryInfo, fileIndex }) {
console.log(`Unrestricting Putio ${infoHash} [${fileIndex}]`);
const Putio = createPutioAPI(apiKey)
return _resolve(Putio, infoHash, cachedEntryInfo, fileIndex)
.catch(error => {
if (error?.data?.status_code === 401) {
console.log(`Access denied to Putio ${infoHash} [${fileIndex}]`);
return StaticResponse.FAILED_ACCESS;
}
return Promise.reject(`Failed Putio adding torrent ${JSON.stringify(error.data || error)}`);
});
}
async function _resolve(Putio, infoHash, cachedEntryInfo, fileIndex) {
if (infoHash === 'null') {
return _unrestrictVideo(Putio, fileIndex);
}
const torrent = await _createOrFindTorrent(Putio, infoHash);
if (torrent && statusReady(torrent.status)) {
return _unrestrictLink(Putio, torrent, cachedEntryInfo, fileIndex);
} else if (torrent && statusDownloading(torrent.status)) {
console.log(`Downloading to Putio ${infoHash} [${fileIndex}]...`);
return StaticResponse.DOWNLOADING;
} else if (torrent && statusError(torrent.status)) {
console.log(`Retrying downloading to Putio ${infoHash} [${fileIndex}]...`);
return _retryCreateTorrent(Putio, infoHash, cachedEntryInfo, fileIndex);
}
return Promise.reject("Failed Putio adding torrent");
}
async function _createOrFindTorrent(Putio, infoHash) {
return _findTorrent(Putio, infoHash)
.catch(() => _createTorrent(Putio, infoHash));
}
async function _retryCreateTorrent(Putio, infoHash, encodedFileName, fileIndex) {
const newTorrent = await _createTorrent(Putio, infoHash);
return newTorrent && statusReady(newTorrent.status)
? _unrestrictLink(Putio, newTorrent, encodedFileName, fileIndex)
: StaticResponse.FAILED_DOWNLOAD;
}
async function _findTorrent(Putio, infoHash) {
const torrents = await Putio.Transfers.Query().then(response => response.data.transfers);
const foundTorrents = torrents.filter(torrent => torrent.source.toLowerCase().includes(infoHash));
const nonFailedTorrent = foundTorrents.find(torrent => !statusError(torrent.status));
const foundTorrent = nonFailedTorrent || foundTorrents[0];
if (foundTorrents && !foundTorrents.userfile_exists) {
return await Putio.Transfers.Cancel(foundTorrents.id).then(() => Promise.reject())
}
return foundTorrent || Promise.reject('No recent torrent found in Putio');
}
async function _findInfoHash(Putio, fileId) {
const torrents = await Putio.Transfers.Query().then(response => response?.data?.transfers);
const foundTorrent = torrents.find(torrent => `${torrent.file_id}` === fileId);
return foundTorrent?.source ? decode(foundTorrent.source).infoHash : undefined;
}
async function _createTorrent(Putio, infoHash) {
const magnetLink = await getMagnetLink(infoHash);
// Add the torrent and then delay for 3 secs for putio to process it and then check it's status.
return Putio.Transfers.Add({ url: magnetLink })
.then(response => _getNewTorrent(Putio, response.data.transfer.id));
}
async function _getNewTorrent(Putio, torrentId, pollCounter = 0, pollRate = 2000, maxPollNumber = 15) {
return Putio.Transfers.Get(torrentId)
.then(response => response.data.transfer)
.then(torrent => statusProcessing(torrent.status) && pollCounter < maxPollNumber
? delay(pollRate).then(() => _getNewTorrent(Putio, torrentId, pollCounter + 1))
: torrent);
}
async function _unrestrictLink(Putio, torrent, encodedFileName, fileIndex) {
const targetVideo = await _getTargetFile(Putio, torrent, encodedFileName, fileIndex);
return _unrestrictVideo(Putio, targetVideo.id);
}
async function _unrestrictVideo(Putio, videoId) {
const response = await Putio.File.GetStorageURL(videoId);
const downloadUrl = response.data.url
console.log(`Unrestricted Putio [${videoId}] to ${downloadUrl}`);
return downloadUrl;
}
async function _getTargetFile(Putio, torrent, encodedFileName, fileIndex) {
const targetFileName = decodeURIComponent(encodedFileName);
let targetFile;
let files = await _getFiles(Putio, torrent.file_id);
let videos = [];
while (!targetFile && files.length) {
const folders = files.filter(file => file.file_type === 'FOLDER');
videos = videos.concat(files.filter(file => isVideo(file.name)));
// when specific file index is defined search by filename
// when it's not defined find all videos and take the largest one
targetFile = Number.isInteger(fileIndex)
? videos.find(video => sameFilename(targetFileName, video.name))
: !folders.length && videos.sort((a, b) => b.size - a.size)[0];
files = !targetFile
? await Promise.all(folders.map(folder => _getFiles(Putio, folder.id)))
.then(results => results.reduce((a, b) => a.concat(b), []))
: [];
}
return targetFile || Promise.reject(`No target file found for Putio [${torrent.hash}] ${targetFileName}`);
}
async function _getFiles(Putio, fileId) {
const response = await Putio.Files.Query(fileId)
.catch(error => Promise.reject({ ...error.data, path: error.request.path }));
return response.data.files.length
? response.data.files
: [response.data.parent];
}
function createPutioAPI(apiKey) {
const clientId = apiKey.replace(/@.*/, '');
const token = apiKey.replace(/.*@/, '');
const Putio = new PutioAPI({ clientID: clientId });
Putio.setToken(token);
return Putio;
}
function statusError(status) {
return ['ERROR'].includes(status);
}
function statusDownloading(status) {
return ['WAITING', 'IN_QUEUE', 'DOWNLOADING'].includes(status);
}
function statusProcessing(status) {
return ['WAITING', 'IN_QUEUE', 'COMPLETING'].includes(status);
}
function statusReady(status) {
return ['COMPLETED', 'SEEDING'].includes(status);
}

View File

@@ -0,0 +1,399 @@
import RealDebridClient from 'real-debrid-api';
import { cacheAvailabilityResults, getCachedAvailabilityResults } from '../lib/cache.js';
import { isVideo, isArchive } from '../lib/extension.js';
import { getMagnetLink } from '../lib/magnetHelper.js';
import { delay } from '../lib/promises.js';
import { Type } from '../lib/types.js';
import { chunkArray, BadTokenError, AccessDeniedError } from './mochHelper.js';
import StaticResponse from './static.js';
const MIN_SIZE = 5 * 1024 * 1024; // 5 MB
const CATALOG_MAX_PAGE = 1;
const CATALOG_PAGE_SIZE = 100;
const NON_BLACKLIST_ERRORS = ['ESOCKETTIMEDOUT', 'EAI_AGAIN', '504 Gateway Time-out'];
const KEY = 'realdebrid';
const DEBRID_DOWNLOADS = 'Downloads';
export async function getCachedStreams(streams, apiKey) {
const hashes = streams.map(stream => stream.infoHash);
const available = await _getInstantAvailable(hashes, apiKey);
return available && streams
.reduce((mochStreams, stream) => {
const cachedEntry = available[stream.infoHash];
const cachedIds = _getCachedFileIds(stream.fileIdx, cachedEntry);
mochStreams[stream.infoHash] = {
url: `${apiKey}/${stream.infoHash}/null/${stream.fileIdx}`,
cached: !!cachedIds.length
};
return mochStreams;
}, {})
}
async function _getInstantAvailable(hashes, apiKey, retries = 3, maxChunkSize = 150) {
const cachedResults = await getCachedAvailabilityResults(hashes);
const missingHashes = hashes.filter(infoHash => !cachedResults[infoHash]);
if (!missingHashes.length) {
return cachedResults
}
const options = await getDefaultOptions();
const RD = new RealDebridClient(apiKey, options);
const hashBatches = chunkArray(missingHashes, maxChunkSize)
return Promise.all(hashBatches.map(batch => RD.torrents.instantAvailability(batch)
.then(response => {
if (typeof response !== 'object') {
return Promise.reject(new Error('RD returned non JSON response: ' + response));
}
return processAvailabilityResults(response);
})))
.then(results => results.reduce((all, result) => Object.assign(all, result), {}))
.then(results => cacheAvailabilityResults(results))
.then(results => Object.assign(cachedResults, results))
.catch(error => {
if (toCommonError(error)) {
return Promise.reject(error);
}
if (!error && maxChunkSize !== 1) {
// sometimes due to large response size RD responds with an empty body. Reduce chunk size to reduce body
console.log(`Reducing chunk size for availability request: ${hashes[0]}`);
return _getInstantAvailable(hashes, apiKey, retries - 1, Math.ceil(maxChunkSize / 10));
}
if (retries > 0 && NON_BLACKLIST_ERRORS.some(v => error?.message?.includes(v))) {
return _getInstantAvailable(hashes, apiKey, retries - 1);
}
console.warn(`Failed RealDebrid cached [${hashes[0]}] torrent availability request:`, error.message);
return undefined;
});
}
function processAvailabilityResults(availabilityResults) {
const processedResults = {};
Object.entries(availabilityResults)
.forEach(([infoHash, hosterResults]) => processedResults[infoHash] = getCachedIds(hosterResults));
return processedResults;
}
function getCachedIds(hosterResults) {
if (!hosterResults || Array.isArray(hosterResults)) {
return [];
}
// if not all cached files are videos, then the torrent will be zipped to a rar
return Object.values(hosterResults)
.reduce((a, b) => a.concat(b), [])
.filter(cached => Object.keys(cached).length && Object.values(cached).every(file => isVideo(file.filename)))
.map(cached => Object.keys(cached))
.sort((a, b) => b.length - a.length)
.filter((cached, index, array) => index === 0 || cached.some(id => !array[0].includes(id)));
}
function _getCachedFileIds(fileIndex, cachedResults) {
if (!cachedResults || !Array.isArray(cachedResults)) {
return [];
}
const cachedIds = Number.isInteger(fileIndex)
? cachedResults.find(ids => ids.includes(`${fileIndex + 1}`))
: cachedResults[0];
return cachedIds || [];
}
export async function getCatalog(apiKey, offset, ip) {
if (offset > 0) {
return [];
}
const options = await getDefaultOptions(ip);
const RD = new RealDebridClient(apiKey, options);
const downloadsMeta = {
id: `${KEY}:${DEBRID_DOWNLOADS}`,
type: Type.OTHER,
name: DEBRID_DOWNLOADS
};
const torrentMetas = await _getAllTorrents(RD)
.then(torrents => Array.isArray(torrents) ? torrents : [])
.then(torrents => torrents
.filter(torrent => torrent && statusReady(torrent.status))
.map(torrent => ({
id: `${KEY}:${torrent.id}`,
type: Type.OTHER,
name: torrent.filename
})));
return [downloadsMeta].concat(torrentMetas)
}
export async function getItemMeta(itemId, apiKey, ip) {
const options = await getDefaultOptions(ip);
const RD = new RealDebridClient(apiKey, options);
if (itemId === DEBRID_DOWNLOADS) {
const videos = await _getAllDownloads(RD)
.then(downloads => downloads
.map(download => ({
id: `${KEY}:${DEBRID_DOWNLOADS}:${download.id}`,
// infoHash: allTorrents
// .filter(torrent => (torrent.links || []).find(link => link === download.link))
// .map(torrent => torrent.hash.toLowerCase())[0],
title: download.filename,
released: new Date(download.generated).toISOString(),
streams: [{ url: download.download }]
})));
return {
id: `${KEY}:${DEBRID_DOWNLOADS}`,
type: Type.OTHER,
name: DEBRID_DOWNLOADS,
videos: videos
};
}
return _getTorrentInfo(RD, itemId)
.then(torrent => ({
id: `${KEY}:${torrent.id}`,
type: Type.OTHER,
name: torrent.filename,
infoHash: torrent.hash.toLowerCase(),
videos: torrent.files
.filter(file => file.selected)
.filter(file => isVideo(file.path))
.map((file, index) => ({
id: `${KEY}:${torrent.id}:${file.id}`,
title: file.path,
released: new Date(new Date(torrent.added).getTime() - index).toISOString(),
streams: [{ url: `${apiKey}/${torrent.hash.toLowerCase()}/null/${file.id - 1}` }]
}))
}))
}
async function _getAllTorrents(RD, page = 1) {
return RD.torrents.get(page - 1, page, CATALOG_PAGE_SIZE)
.then(torrents => torrents && torrents.length === CATALOG_PAGE_SIZE && page < CATALOG_MAX_PAGE
? _getAllTorrents(RD, page + 1)
.then(nextTorrents => torrents.concat(nextTorrents))
.catch(() => torrents)
: torrents)
}
async function _getAllDownloads(RD, page = 1) {
return RD.downloads.get(page - 1, page, CATALOG_PAGE_SIZE);
}
export async function resolve({ ip, isBrowser, apiKey, infoHash, fileIndex }) {
console.log(`Unrestricting RealDebrid ${infoHash} [${fileIndex}]`);
const options = await getDefaultOptions(ip);
const RD = new RealDebridClient(apiKey, options);
const cachedFileIds = await _resolveCachedFileIds(infoHash, fileIndex, apiKey);
return _resolve(RD, infoHash, cachedFileIds, fileIndex, isBrowser)
.catch(error => {
if (accessDeniedError(error)) {
console.log(`Access denied to RealDebrid ${infoHash} [${fileIndex}]`);
return StaticResponse.FAILED_ACCESS;
}
if (infringingFile(error)) {
console.log(`Infringing file removed from RealDebrid ${infoHash} [${fileIndex}]`);
return StaticResponse.FAILED_INFRINGEMENT;
}
return Promise.reject(`Failed RealDebrid adding torrent ${JSON.stringify(error)}`);
});
}
async function _resolveCachedFileIds(infoHash, fileIndex, apiKey) {
const available = await _getInstantAvailable([infoHash], apiKey);
const cachedEntry = available?.[infoHash];
const cachedIds = _getCachedFileIds(fileIndex, cachedEntry);
return cachedIds?.join(',');
}
async function _resolve(RD, infoHash, cachedFileIds, fileIndex, isBrowser) {
const torrentId = await _createOrFindTorrentId(RD, infoHash, cachedFileIds, fileIndex);
const torrent = await _getTorrentInfo(RD, torrentId);
if (torrent && statusReady(torrent.status)) {
return _unrestrictLink(RD, torrent, fileIndex, isBrowser);
} else if (torrent && statusDownloading(torrent.status)) {
console.log(`Downloading to RealDebrid ${infoHash} [${fileIndex}]...`);
return StaticResponse.DOWNLOADING;
} else if (torrent && statusMagnetError(torrent.status)) {
console.log(`Failed RealDebrid opening torrent ${infoHash} [${fileIndex}] due to magnet error`);
return StaticResponse.FAILED_OPENING;
} else if (torrent && statusError(torrent.status)) {
return _retryCreateTorrent(RD, infoHash, fileIndex);
} else if (torrent && (statusWaitingSelection(torrent.status) || statusOpening(torrent.status))) {
console.log(`Trying to select files on RealDebrid ${infoHash} [${fileIndex}]...`);
return _selectTorrentFiles(RD, torrent)
.then(() => {
console.log(`Downloading to RealDebrid ${infoHash} [${fileIndex}]...`);
return StaticResponse.DOWNLOADING
})
.catch(error => {
console.log(`Failed RealDebrid opening torrent ${infoHash} [${fileIndex}]:`, error);
return StaticResponse.FAILED_OPENING;
});
}
return Promise.reject(`Failed RealDebrid adding torrent ${JSON.stringify(torrent)}`);
}
async function _createOrFindTorrentId(RD, infoHash, cachedFileIds, fileIndex) {
return _findTorrent(RD, infoHash, fileIndex)
.catch(() => _createTorrentId(RD, infoHash, cachedFileIds));
}
async function _findTorrent(RD, infoHash, fileIndex) {
const torrents = await RD.torrents.get(0, 1) || [];
const foundTorrents = torrents
.filter(torrent => torrent.hash.toLowerCase() === infoHash)
.filter(torrent => !statusError(torrent.status));
const foundTorrent = await _findBestFitTorrent(RD, foundTorrents, fileIndex);
return foundTorrent?.id || Promise.reject('No recent torrent found');
}
async function _findBestFitTorrent(RD, torrents, fileIndex) {
if (torrents.length === 1) {
return torrents[0];
}
const torrentInfos = await Promise.all(torrents.map(torrent => _getTorrentInfo(RD, torrent.id)));
const bestFitTorrents = torrentInfos
.filter(torrent => torrent.files.find(f => f.id === fileIndex + 1 && f.selected))
.sort((a, b) => b.links.length - a.links.length);
return bestFitTorrents[0] || torrents[0];
}
async function _getTorrentInfo(RD, torrentId) {
if (!torrentId || typeof torrentId === 'object') {
return torrentId || Promise.reject('No RealDebrid torrentId provided')
}
return RD.torrents.info(torrentId);
}
async function _createTorrentId(RD, infoHash, cachedFileIds) {
const magnetLink = await getMagnetLink(infoHash);
const addedMagnet = await RD.torrents.addMagnet(magnetLink);
if (cachedFileIds && !['null', 'undefined'].includes(cachedFileIds)) {
await RD.torrents.selectFiles(addedMagnet.id, cachedFileIds);
}
return addedMagnet.id;
}
async function _recreateTorrentId(RD, infoHash, fileIndex) {
const newTorrentId = await _createTorrentId(RD, infoHash);
await _selectTorrentFiles(RD, { id: newTorrentId }, fileIndex);
return newTorrentId;
}
async function _retryCreateTorrent(RD, infoHash, fileIndex) {
console.log(`Retry failed download in RealDebrid ${infoHash} [${fileIndex}]...`);
const newTorrentId = await _recreateTorrentId(RD, infoHash, fileIndex);
const newTorrent = await _getTorrentInfo(RD, newTorrentId);
return newTorrent && statusReady(newTorrent.status)
? _unrestrictLink(RD, newTorrent, fileIndex)
: StaticResponse.FAILED_DOWNLOAD;
}
async function _selectTorrentFiles(RD, torrent, fileIndex) {
torrent = statusWaitingSelection(torrent.status) ? torrent : await _openTorrent(RD, torrent.id);
if (torrent?.files && statusWaitingSelection(torrent.status)) {
const videoFileIds = Number.isInteger(fileIndex) ? `${fileIndex + 1}` : torrent.files
.filter(file => isVideo(file.path))
.filter(file => file.bytes > MIN_SIZE)
.map(file => file.id)
.join(',');
return RD.torrents.selectFiles(torrent.id, videoFileIds);
}
return Promise.reject('Failed RealDebrid torrent file selection')
}
async function _openTorrent(RD, torrentId, pollCounter = 0, pollRate = 2000, maxPollNumber = 15) {
return _getTorrentInfo(RD, torrentId)
.then(torrent => torrent && statusOpening(torrent.status) && pollCounter < maxPollNumber
? delay(pollRate).then(() => _openTorrent(RD, torrentId, pollCounter + 1))
: torrent);
}
async function _unrestrictLink(RD, torrent, fileIndex, isBrowser) {
const targetFile = torrent.files.find(file => file.id === fileIndex + 1)
|| torrent.files.filter(file => file.selected).sort((a, b) => b.bytes - a.bytes)[0];
if (!targetFile.selected) {
console.log(`Target RealDebrid file is not downloaded: ${JSON.stringify(targetFile)}`);
await _recreateTorrentId(RD, torrent.hash.toLowerCase(), fileIndex);
return StaticResponse.DOWNLOADING;
}
const selectedFiles = torrent.files.filter(file => file.selected);
const fileLink = torrent.links.length === 1
? torrent.links[0]
: torrent.links[selectedFiles.indexOf(targetFile)];
if (!fileLink?.length) {
console.log(`No RealDebrid links found for ${torrent.hash} [${fileIndex}]`);
return _retryCreateTorrent(RD, torrent.hash, fileIndex)
}
return _unrestrictFileLink(RD, fileLink, torrent, fileIndex, isBrowser);
}
async function _unrestrictFileLink(RD, fileLink, torrent, fileIndex, isBrowser) {
return RD.unrestrict.link(fileLink)
.then(response => {
if (isArchive(response.download)) {
if (torrent.files.filter(file => file.selected).length > 1) {
return _retryCreateTorrent(RD, torrent.hash, fileIndex)
}
return StaticResponse.FAILED_RAR;
}
// if (isBrowser && response.streamable) {
// return RD.streaming.transcode(response.id)
// .then(streamResponse => streamResponse.apple.full)
// }
return response.download;
})
.then(unrestrictedLink => {
console.log(`Unrestricted RealDebrid ${torrent.hash} [${fileIndex}] to ${unrestrictedLink}`);
return unrestrictedLink;
})
.catch(error => {
if (error.code === 19) {
return _retryCreateTorrent(RD, torrent.hash.toLowerCase(), fileIndex);
}
return Promise.reject(error);
});
}
export function toCommonError(error) {
if (error && error.code === 8) {
return BadTokenError;
}
if (error && accessDeniedError(error)) {
return AccessDeniedError;
}
return undefined;
}
function statusError(status) {
return ['error', 'magnet_error'].includes(status);
}
function statusMagnetError(status) {
return status === 'magnet_error';
}
function statusOpening(status) {
return status === 'magnet_conversion';
}
function statusWaitingSelection(status) {
return status === 'waiting_files_selection';
}
function statusDownloading(status) {
return ['downloading', 'uploading', 'queued'].includes(status);
}
function statusReady(status) {
return ['downloaded', 'dead'].includes(status);
}
function accessDeniedError(error) {
return [9, 20].includes(error?.code);
}
function infringingFile(error) {
return error && error.code === 35;
}
async function getDefaultOptions(ip) {
return { ip, timeout: 10000 };
}

View File

@@ -0,0 +1,16 @@
const staticVideoUrls = {
DOWNLOADING: `videos/downloading_v2.mp4`,
FAILED_DOWNLOAD: `videos/download_failed_v2.mp4`,
FAILED_ACCESS: `videos/failed_access_v2.mp4`,
FAILED_RAR: `videos/failed_rar_v2.mp4`,
FAILED_OPENING: `videos/failed_opening_v2.mp4`,
FAILED_UNEXPECTED: `videos/failed_unexpected_v2.mp4`,
FAILED_INFRINGEMENT: `videos/failed_infringement_v2.mp4`
}
export function isStaticUrl(url) {
return Object.values(staticVideoUrls).some(videoUrl => url?.endsWith(videoUrl));
}
export default staticVideoUrls

122
src/addon/src/serverless.js Normal file
View File

@@ -0,0 +1,122 @@
import cors from 'cors';
import rateLimit from "express-rate-limit";
import qs from 'querystring';
import requestIp from 'request-ip';
import Router from 'router';
import userAgentParser from 'ua-parser-js';
import addonInterface from './addon.js';
import { parseConfiguration } from './lib/configuration.js';
import landingTemplate from './lib/landingTemplate.js';
import { manifest } from './lib/manifest.js';
import * as moch from './moch/moch.js';
import { computeTorrentStatistics } from './lib/repository.js';
const router = new Router();
const limiter = rateLimit({
windowMs: 60 * 60 * 1000, // 1 hour
max: 3000000, // limit each IP to 300 requests per windowMs
headers: false,
keyGenerator: (req) => requestIp.getClientIp(req)
})
router.use(cors())
router.get('/', (_, res) => {
res.redirect('/configure')
res.end();
});
router.get('/:configuration?/configure', (req, res) => {
const configValues = parseConfiguration(req.params.configuration || '');
const landingHTML = landingTemplate(manifest(configValues), configValues);
res.setHeader('content-type', 'text/html');
res.end(landingHTML);
});
router.get('/:configuration?/manifest.json', (req, res) => {
const configValues = parseConfiguration(req.params.configuration || '');
const manifestBuf = JSON.stringify(manifest(configValues));
res.setHeader('Content-Type', 'application/json; charset=utf-8');
res.end(manifestBuf)
});
router.get('/:configuration?/:resource/:type/:id/:extra?.json', limiter, (req, res, next) => {
const { configuration, resource, type, id } = req.params;
const extra = req.params.extra ? qs.parse(req.url.split('/').pop().slice(0, -5)) : {}
const ip = requestIp.getClientIp(req);
const host = `${req.protocol}://${req.headers.host}`;
const configValues = { ...extra, ...parseConfiguration(configuration), id, type, ip, host };
addonInterface.get(resource, type, id, configValues)
.then(resp => {
const cacheHeaders = {
cacheMaxAge: 'max-age',
staleRevalidate: 'stale-while-revalidate',
staleError: 'stale-if-error'
};
const cacheControl = Object.keys(cacheHeaders)
.map(prop => Number.isInteger(resp[prop]) && cacheHeaders[prop] + '=' + resp[prop])
.filter(val => !!val).join(', ');
res.setHeader('Cache-Control', `${cacheControl}, public`);
res.setHeader('Content-Type', 'application/json; charset=utf-8');
res.end(JSON.stringify(resp));
})
.catch(err => {
if (err.noHandler) {
if (next) {
next()
} else {
res.writeHead(404);
res.end(JSON.stringify({ err: 'not found' }));
}
} else {
console.error(err);
res.writeHead(500);
res.end(JSON.stringify({ err: 'handler error' }));
}
});
});
router.get('/:moch/:apiKey/:infoHash/:cachedEntryInfo/:fileIndex/:filename?', (req, res) => {
const userAgent = req.headers['user-agent'] || '';
const parameters = {
mochKey: req.params.moch,
apiKey: req.params.apiKey,
infoHash: req.params.infoHash.toLowerCase(),
fileIndex: isNaN(req.params.fileIndex) ? undefined : parseInt(req.params.fileIndex),
cachedEntryInfo: req.params.cachedEntryInfo,
ip: requestIp.getClientIp(req),
host: `${req.protocol}://${req.headers.host}`,
isBrowser: !userAgent.includes('Stremio') && !!userAgentParser(userAgent).browser.name
}
moch.resolve(parameters)
.then(url => {
res.writeHead(302, { Location: url });
res.end();
})
.catch(error => {
console.log(error);
res.statusCode = 404;
res.end();
});
});
/**
* Temporary solution to provide statistics (NOT FOR PRODUCTION USE)
* easy way to know how many torrents are stored in the PG database
* These data should not be exposed to the public
*/
router.get('/statistics', async (req, res) => {
try {
const stats = await computeTorrentStatistics();
res.json(stats);
} catch (error) {
res.status(500).json({ error: 'Failed to compute statistics' });
}
});
export default function (req, res) {
router(req, res, function () {
res.statusCode = 404;
res.end();
});
}

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.