1629 commits

Author	SHA1	Message	Date
Nick Sweeting	0a5b22700c	bump deps	2024-08-22 15:20:52 -07:00
Nick Sweeting	afe1307617	fix created_by field migration to create User properly if none exists	2024-08-22 15:20:36 -07:00
Nick Sweeting	9b1659c72f	make created_by_id autoapply to any ArchiveResults created under Snapshot	2024-08-20 19:43:07 -07:00
Nick Sweeting	c30ae1d2cb	add created_by_id to all Snapshot creation functions	2024-08-20 19:28:28 -07:00
Nick Sweeting	aa282daadf	show CONSTANTS in separate section of django admin config	2024-08-20 19:27:21 -07:00
Nick Sweeting	ea81f2fc14	use ABIDAdmin as base ModelAdmin	2024-08-20 19:26:40 -07:00
Nick Sweeting	57e48b432c	show redacted API tokens in django admin	2024-08-20 19:25:16 -07:00
Nick Sweeting	ff619fd31f	add api migrations	2024-08-20 18:31:40 -07:00
Nick Sweeting	0285aa52a0	config and attr access improvements	2024-08-20 18:31:21 -07:00
Nick Sweeting	4ae186dfca	fix ABID generation consistency when self._state.adding is True	2024-08-20 05:56:19 -07:00
Nick Sweeting	9d2116ad9a	migration to make fields non editable	2024-08-20 03:46:04 -07:00
Nick Sweeting	9c8ece4d7c	minor css and APIToken Admin UI fixes	2024-08-20 03:45:49 -07:00
Nick Sweeting	75018ed10b	Merge branch 'dev' into feat/search-fields	2024-08-20 03:37:56 -07:00
Nick Sweeting	05880dba8b	bump version to 0.8.2	2024-08-20 03:36:07 -07:00
Nick Sweeting	267964881c	fix tags editor in Snapshot list view	2024-08-20 03:29:57 -07:00
Nick Sweeting	849b4963a1	add migrations	2024-08-20 01:58:44 -07:00
Nick Sweeting	52a813aa80	fix title display in admin UI and abid filter matching in urls	2024-08-20 01:58:36 -07:00
Nick Sweeting	9273db528e	fix abid generation migrations to be historically consistent	2024-08-20 01:58:19 -07:00
Nick Sweeting	506b3d28d4	fix admin UI TagInline and ArchiveResultInline form POST handling	2024-08-20 01:57:07 -07:00
Nick Sweeting	54acfd9f86	improve REST API filter parameters and pagination	2024-08-20 01:56:37 -07:00
Nick Sweeting	850448b42c	add salt args in more places	2024-08-20 01:56:07 -07:00
Nick Sweeting	c4ef2993b2	update REST API and Admin UI to use new id and old_id exclusively	2024-08-19 20:00:04 -07:00
Nick Sweeting	344e902fc6	migrate SnapshotTag to use new snapshot id	2024-08-19 19:42:25 -07:00
Nick Sweeting	74c11d41be	dont allow modifying Snapshot.timestamp as it would break folder dirs	2024-08-19 19:42:01 -07:00
Nick Sweeting	c68a66f74e	add salt kwarg to abid generation funcs	2024-08-19 19:41:43 -07:00
Nick Sweeting	a49739b41c	add hacky workaround for TagInline not showing in admin Snapshot change view	2024-08-19 18:36:35 -07:00
Nick Sweeting	cf2faecf61	add migrations for SnapshotTag through model	2024-08-19 18:36:20 -07:00
Nick Sweeting	7164fb961c	hotlink to snapshots using generic search query link instead of id__startswith	2024-08-19 18:35:35 -07:00
Nick Sweeting	3148d2a3ef	add squashmigrations to allowed mgmgt command list	2024-08-19 18:35:07 -07:00
Nick Sweeting	2c157f0b5b	increase max POST field limit for admin	2024-08-19 18:34:52 -07:00
Nick Sweeting	18ebaed7ed	fix admin UI display of tags	2024-08-19 18:34:01 -07:00
Nick Sweeting	57d31b2b14	fix snapshot uuid	2024-08-18 01:07:21 -07:00
Nick Sweeting	8c50257fe9	move snapshot id to old_id	2024-08-18 00:24:38 -07:00
Nick Sweeting	951025228f	add tag endpoint	2024-08-18 00:24:14 -07:00
Nick Sweeting	a271bcb4ce	use new ids for ArchiveResult API	2024-08-17 23:03:43 -07:00
Nick Sweeting	f72debfdb2	migrate ArchiveResult.id to old_id, and make uuid main id	2024-08-17 22:58:35 -07:00
Nick Sweeting	033ec08d0c	save snapshot ids during migration	2024-08-17 21:56:45 -07:00
Nick Sweeting	4d0bbfccfc	add uri salt and fix api url namespaces	2024-08-17 21:56:23 -07:00
Nick Sweeting	fba3995d86	reorder version output	2024-08-17 19:38:51 -07:00
Nick Sweeting	6a33de6715	remove archive.today link and add JSON	2024-08-17 19:31:43 -07:00
Nick Sweeting	3a87a7fb8c	re-arrange snapshot detail page preference and add opus support	2024-08-17 19:31:34 -07:00
Nick Sweeting	5e0cc926f1	show ulid in archivebox admin	2024-08-17 19:31:18 -07:00
Nick Sweeting	b7745ae37a	add uri salt when creating ABID	2024-08-17 19:30:58 -07:00
Lucas Schwiderski	10f0c4a077	Implement searching individual fields ... So far only for the public view, since we already have a custom search form there, where this is easy to add. This initial implementation supports the common set of metadata fields that the searchbar placeholder also mentions, but adding more fields is trivial.	2024-06-19 16:50:16 +02:00
Nick Sweeting	99e6f0c93f	bump versions	2024-06-04 04:17:33 -07:00
Nick Sweeting	de489d3c60	minor snapshot details ui fixes and migrations log msg improvements	2024-06-04 04:17:32 -07:00
Nick Sweeting	f574d34357	wrap migrations maker in try catch	2024-06-03 03:02:00 -07:00
Nick Sweeting	c63917a22d	fix more CHROME_USER_DATA_DIR issues	2024-06-03 02:51:11 -07:00
Nick Sweeting	1cd62ecc61	add filesizes and stray files in snapshot dir to snapshot_live ui	2024-06-03 02:31:35 -07:00
Nick Sweeting	729f05ab93	fix snapshot_live iframe scrollTo not working due to origin sandbox rules	2024-06-02 19:15:12 -07:00
Nick Sweeting	11a24d9640	merge migrations between branches	2024-06-02 19:14:58 -07:00
Nick Sweeting	b3f2a71c93	bump requirements and version to 0.8.1	2024-06-02 19:14:45 -07:00
Nick Sweeting	07d80db5aa	fix CHROME_USER_DATA_DIR split str error	2024-06-02 19:14:20 -07:00
Nick Sweeting	f72bae8eec	autodiscover plugins on startup	2024-05-17 20:48:11 -07:00
Nick Sweeting	48becde9b4	add new pydantic-based plugin system	2024-05-17 20:13:54 -07:00
Nick Sweeting	e4176dbf7a	change live snapshot preview iframe sandbox rules	2024-05-17 20:12:37 -07:00
Nick Sweeting	774ce3fda7	fix singlefile extractor exception when result is none	2024-05-17 20:12:18 -07:00
Nick Sweeting	29c7aa26bc	show original section titles in config admin ui	2024-05-17 20:11:54 -07:00
Nick Sweeting	acfd346440	make abids searchable in the admin ui	2024-05-17 20:11:31 -07:00
Nick Sweeting	a1afd0211f	fix abid calculation	2024-05-17 20:11:00 -07:00
Nick Sweeting	fdf6f465db	add docstrings	2024-05-14 18:46:38 -07:00
Nick Sweeting	406f57031a	add API support for obj.pk .uuid .abid	2024-05-13 08:52:02 -07:00
Nick Sweeting	a4cc10d7f8	add migrations for third round of field changes	2024-05-13 07:50:22 -07:00
Nick Sweeting	241a7c6ab2	add created, modified, updated, created_by and update django admin	2024-05-13 07:50:07 -07:00
Nick Sweeting	1ba8215072	only use domain part of uri for hash	2024-05-13 07:49:36 -07:00
Nick Sweeting	206e7e74b3	add migrations to create and populate ABIDField and UUIDField values	2024-05-13 05:13:42 -07:00
Nick Sweeting	0420662174	switch everywhere to use Snapshot.pk and ArchiveResult.pk instead of id	2024-05-13 05:12:12 -07:00
Nick Sweeting	9733b8d04c	remove accidentally commited db	2024-05-13 02:38:02 -07:00
Nick Sweeting	4f9f22e024	create abid_utils with new ABID type for ArchiveBox IDs	2024-05-13 02:37:48 -07:00
Nick Sweeting	f896e5dbeb	switch from monkey patching WebhookModel to using swappable	2024-05-13 02:36:15 -07:00
Nick Sweeting	e97d779cd3	move monkey patches to dedicated file	2024-05-13 02:35:19 -07:00
Nick Sweeting	b5ad134264	dont wait for ipython history saver thread before shell exit	2024-05-12 19:25:55 -07:00
Nick Sweeting	ce833e8ead	automatically create storage directories and symlinks based on ulid	2024-05-12 05:21:58 -07:00
Nick Sweeting	33bc4622a0	add ulid and typeid to Snapshot and ArchiveResult	2024-05-12 04:45:34 -07:00
Nick Sweeting	0529099639	create cache dir and sqlite db	2024-05-12 01:42:34 -07:00
Nick Sweeting	e1a04729b3	wait for bg threads to finish before exiting	2024-05-12 01:42:20 -07:00
Nick Sweeting	0438924491	remove unused template vars	2024-05-11 22:43:26 -07:00
Nick Sweeting	2b4b6e5b3a	select single snapshot row automatically when hotlinking to admin actions	2024-05-11 22:33:39 -07:00
Nick Sweeting	27809f2976	fix python3.10 nested fstring quotes	2024-05-11 22:33:27 -07:00
Nick Sweeting	baf24d2d6a	change wording on 404 snapshot detail page	2024-05-11 22:33:17 -07:00
Nick Sweeting	8841e8b181	add new live django template for snapshot detail page	2024-05-11 22:33:02 -07:00
Nick Sweeting	457c42bf84	load EXTRACTORS dynamically using importlib.import_module	2024-05-11 22:28:59 -07:00
Nick Sweeting	c7f55fc3ba	make CHROME_USER_DATA_DIR error quieter	2024-05-11 22:16:10 -07:00
Nick Sweeting	102e87578c	re-lock packages for python3.10 because we have to support it still	2024-05-11 15:06:19 -07:00
Nick Sweeting	913590ee39	explain weird use of ellipses magic value	2024-05-11 15:02:43 -07:00
Nick Sweeting	b90ba6c909	change header auth to use X-ArchiveBox-API-Key so it doesnt conflict with other auth headers	2024-05-08 20:02:47 -07:00
Nick Sweeting	4c5a3fba8b	more fixes for wget_output_path	2024-05-07 05:38:29 -07:00
Nick Sweeting	f2729c9dc7	revert wget character limit back to windows, ascii is allows MORE not less	2024-05-07 05:24:16 -07:00
Nick Sweeting	9b21ce490e	add workaround logic to catch paths that are too long or contain unprintable characters	2024-05-07 05:03:23 -07:00
Nick Sweeting	f62cb5fb43	change wget to use stricter ascii filepath normalization	2024-05-07 05:03:01 -07:00
Nick Sweeting	f770bba3cf	fix OSError 36 caused by checking for path that is too long to exist	2024-05-07 04:12:07 -07:00
Nick Sweeting	ef856e8051	Merge branch 'dev' into issue1316	2024-05-06 23:14:16 -07:00
Nick Sweeting	27d5d1ddc8	revert queryset intersection back to union for search results	2024-05-06 23:13:52 -07:00
Nick Sweeting	f472705d10	Change phrasing (#1419)	2024-05-06 23:11:26 -07:00
Nick Sweeting	3095265880	fix inner quote not escaped	2024-05-06 23:10:59 -07:00
Brandl	8ccd606973	Fix quotation ... Fixes: => ERROR [stage-0 22/23] RUN "/app"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt 1.7s ------ > [stage-0 22/23] RUN "/app"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt: 1.665 Traceback (most recent call last): 1.665 File "/usr/local/bin/archivebox", line 5, in <module> 1.665 from archivebox.cli import main 1.665 File "/app/archivebox/cli/__init__.py", line 83, in <module> 1.665 SUBCOMMANDS = list_subcommands() 1.665 ^^^^^^^^^^^^^^^^^^ 1.665 File "/app/archivebox/cli/__init__.py", line 43, in list_subcommands 1.665 module = import_module('.archivebox_{}'.format(subcommand), __package__) 1.665 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1.665 File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module 1.665 return _bootstrap._gcd_import(name[level:], package, level) 1.665 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1.666 File "/app/archivebox/cli/archivebox_add.py", line 11, in <module> 1.666 from ..main import add 1.666 File "/app/archivebox/main.py", line 233 1.666 f'COMMIT_HASH={COMMIT_HASH[:7] if COMMIT_HASH else 'unknown'}', 1.666 ^^^^^^^ 1.666 SyntaxError: f-string: expecting '}'	2024-05-06 21:04:14 +02:00
Nick Sweeting	027c029316	redact passwords, keys, and secret tokens in admin UI	2024-05-06 11:06:42 -07:00
Nick Sweeting	8667ed29f1	improve API webhooks helptext and change app_label to API	2024-05-06 08:11:01 -07:00
Evan Boehs	f998647350	change phrasing	2024-05-06 10:32:36 -04:00
Nick Sweeting	d782bafe2e	fix storages missing stackfiles error	2024-05-06 07:14:01 -07:00
Nick Sweeting	47666ec26b	show webhooks config in django admin	2024-05-06 07:13:54 -07:00
Nick Sweeting	f067451267	fix django timezone.utc removed in 5.0	2024-05-06 07:13:25 -07:00
Nick Sweeting	c7fc9c004f	add django-signal-webhooks	2024-05-06 06:58:03 -07:00
Nick Sweeting	3805a1730d	add 0002 api migration	2024-04-30 21:45:02 -07:00
Nick Sweeting	e20eb52f15	fix COMMIT_HASH missing error	2024-04-30 21:43:22 -07:00
Nick Sweeting	18a5b6e99c	move EXTERNAL_LOCATIONS to data locations in version output	2024-04-25 21:36:43 -07:00
Nick Sweeting	6a6ae7468e	fix lint errors	2024-04-25 21:36:11 -07:00
Nick Sweeting	4a5ad32040	add django-requests-tracker	2024-04-25 18:02:01 -07:00
Nick Sweeting	af669d2f37	rename api files for clarity	2024-04-25 05:55:47 -07:00
Nick Sweeting	75153252dc	big overhaul of REST API, split into auth, core, and cli methods	2024-04-25 03:56:22 -07:00
Nick Sweeting	e5aba0dc2e	fix urljoin bug causing multiple slashes to be merged into one	2024-04-24 19:41:31 -07:00
Nick Sweeting	6cb357e76c	fix fix_url_from_markdown assertion to be valid url	2024-04-24 19:41:11 -07:00
Nick Sweeting	128419f991	expand comment about markdown url trailing paren trimming	2024-04-24 17:50:18 -07:00
Nick Sweeting	beb3932d80	replace uses of URL_REGEX with find_all_urls to handle markdown better	2024-04-24 17:45:45 -07:00
Nick Sweeting	c6d644be29	Merge branch 'dev' into issue1316	2024-04-24 16:24:16 -07:00
Nick Sweeting	98c5e69203	bump lockfiles	2024-04-24 14:38:21 -07:00
Nick Sweeting	8dcfa93ec6	Merge branch 'main' into dev	2024-04-24 14:32:07 -07:00
Nick Sweeting	17f40f3ada	Merge branch 'dev' into fix-URL_REGEX	2024-04-23 19:53:58 -07:00
Nick Sweeting	c6f8a33a63	Update util.py	2024-04-23 19:53:18 -07:00
Nick Sweeting	63fc317229	minor pylint fixes in logging_util	2024-04-23 17:45:36 -07:00
Nick Sweeting	756e159dfe	add new bin/lock_pkgs.sh to generate package lockfiles consistently	2024-04-23 17:45:36 -07:00
longzai	e4dc2701ef	fix URL_REGEX 2	2024-04-11 15:51:55 +08:00
Brandl	5f9aac18f2	api v1	2024-04-10 01:29:24 +02:00
longzai	4ae765ec27	fix the URL_REGEX used in generic_html parsers ... Signed-off-by: longzai <437172242@qq.com>	2024-04-08 04:53:05 +08:00
Nick Sweeting	e48159b8a0	cleanup docker-compose by storing crontabs in data dir	2024-03-26 15:24:05 -07:00
Nick Sweeting	ac73fb5129	merge fixes	2024-03-26 15:22:40 -07:00
Nick Sweeting	b4c3aa5097	Merge branch 'main' into dev	2024-03-26 15:01:36 -07:00
Nick Sweeting	a4453b6f87	fix PERSONAS PERSONAS_DIR typo	2024-03-26 14:19:25 -07:00
jim winstead	8b1b01e508	Update to Django 4.2.x, now in LTS until April 2026	2024-03-25 17:46:01 -07:00
Nick Sweeting	c0b5dbcecb	create new data/personas dir to hold cookies and chrome profiles	2024-03-18 14:41:39 -07:00
Nick Sweeting	c5bb99dce1	explicitly use Default profile inside user data dir	2024-03-18 14:40:40 -07:00
Nick Sweeting	1fc5d7c5c8	add USER_AGENT config option to set all USER_AGENTs at once	2024-03-18 14:39:09 -07:00
jim winstead	5478d13d52	Add generic_jsonl parser ... Resolves #1369	2024-03-14 15:42:29 -07:00
Nick Sweeting	ca2c484a8e	Add _EXTRA_ARGS for various extractors (#1360)	2024-03-14 01:55:09 -07:00
Nick Sweeting	48f4b12ae2	Use COOKIES_FILE to fetch page titles (#1364)	2024-03-14 01:52:44 -07:00
Nick Sweeting	099f7d00fe	Use feedparser for RSS parsing (#1362) ... Fixes #1171 Fixes #870 (probably, would need to test against a Wallabag Atom file to Fixes #135 Fixes #123 Fixes #106	2024-03-14 01:51:45 -07:00
Nick Sweeting	3512dc7e60	Disable searching for existing chrome user profiles by default	2024-03-14 00:58:45 -07:00
Ben Muthalaly	f4deb97f59	Add ARGS and EXTRA_ARGS for Mercury extractor	2024-03-05 21:15:38 -06:00
Ben Muthalaly	d8cf09c21e	Remove unnecessary variable length args for dedupe	2024-03-05 21:13:45 -06:00
Ben Muthalaly	5082d61613	Merge branch 'title-cookies-file' of https://github.com/benmuth/ArchiveBox into title-cookies-file	2024-03-05 02:03:03 -06:00
Ben Muthalaly	4686da91e6	Fix cookies being set incorrectly	2024-03-05 01:48:35 -06:00
Naomi Phillips	a729480b75	Add COOKIES_FILE support for singlefile extractor	2024-03-03 02:32:46 -05:00
Ben Muthalaly	d74ddd42ae	Flip dedupe precedence order	2024-03-01 14:50:32 -06:00
jim winstead	9f462a87a8	Use feedparser for RSS parsing in generic_rss and pinboard_rss parsers ... The feedparser packages has 20 years of history and is very good at parsing RSS and Atom, so use that instead of ad-hoc regex and XML parsing. The medium_rss and shaarli_rss parsers weren't touched because they are probably unnecessary. (The special parse for pinboard is just needing because of how tags work.) Doesn't include tests because I haven't figured out how to run them in the docker development setup. Fixes #1171	2024-03-01 11:25:45 -08:00
Nick Sweeting	a577d1ed23	Merge branch 'dev' into title-cookies-file	2024-02-29 21:29:36 -08:00
Nick Sweeting	fe11e1c2f4	check if COOKIE_FILE is file	2024-02-28 18:19:44 -08:00
jim winstead	ccabda4c7d	Handle list of tags in JSON, and be more clever about comma vs. space	2024-02-28 17:38:49 -08:00
jim winstead	178e676e0f	Fix JSON parser by not always mangling the input ... Rather than by assuming the JSON file we are parsing has junk at the beginning (which maybe only used to happen?), try parsing it as-is first, and then fall back to trying again after skipping the first line Fixes #1347	2024-02-27 14:48:19 -08:00
Ben Muthalaly	68326a60ee	Add cookies file to http request in download_url	2024-02-27 15:30:31 -06:00
Ben Muthalaly	4d9c5a7b4b	Add CHROME_EXTRA_ARGS ... Also fix `YOUTUBEDL_EXTRA_ARGS`.	2024-02-23 18:40:03 -06:00