Commit graph

58 commits

Author SHA1 Message Date
Nick Sweeting
5a2c78e14b add proper support for URL_WHITELIST instead of using negation regexes 2021-07-06 23:42:00 -04:00
Nick Sweeting
e4974d3536 support negation patterns by checking both re.search and re.match 2021-07-06 23:17:05 -04:00
Nick Sweeting
36f0646501
Merge pull request #669 from FliegendeWurst/fix-issue-235
add command: --parser option (fixes #235)
2021-03-31 00:53:47 -04:00
FliegendeWurst
60bd9a902e add command: --parser option 2021-03-28 10:09:11 +02:00
Nick Sweeting
33d180afe7 allow filtering snapshots by timestamp in list, update, and remove cmds 2021-02-15 20:48:35 -05:00
Nick Sweeting
d6de04a83a fix lgtm errors 2021-01-30 06:07:35 -05:00
Tim Gates
7bf63d91ff docs: fix simple typo, timstamp -> timestamp
There is a small typo in archivebox/index/__init__.py.

Should read `timestamp` rather than `timstamp`.
2021-01-06 20:03:40 +02:00
Cristian
ce53b0220c refactor: Remove setup_django from index 2020-12-11 17:36:31 -05:00
Cristian
a28547cbca refactor: Remove get_empty_snapshot queryset function and generate it directly 2020-12-11 16:27:15 -05:00
JDC
4eeedae815 Exception handling for indexing and searching 2020-12-06 01:13:39 +02:00
JDC
0f7dba07df feat: add search filter-type to list command 2020-12-06 01:13:37 +02:00
Nick Sweeting
7bc13204e6
Merge branch 'master' into v0.5.0 2020-12-05 17:45:16 -05:00
Hawken Rives
7299b1f5ae
fix "inconsisntencies" typo in error message 2020-12-02 16:28:26 -06:00
Cristian
fa5de72f9f refactor: Move indexing logic out of logging module 2020-11-28 12:34:40 -05:00
Nick Sweeting
c9162a6d09 remove finished/not finished spinners 2020-11-28 01:07:02 -05:00
JDC
cbb3d04c12 Allow list filtering by tag name 2020-11-13 12:06:12 -05:00
Cristian
572b46cecf lint: Remove unused imports 2020-10-23 06:45:56 -05:00
Cristian
ae1484b8bf feat: Remove index.json and index.html generation from the regular process 2020-10-23 06:45:56 -05:00
Angel Rey
ad04fb5300 Replaced os.path in init index 2020-10-02 15:46:39 -05:00
Cristian
b18bbf8874 test: Fix tests post-rebase 2020-09-17 09:09:52 -05:00
apkallum
b99784b919 pathlib with / syntax for config, index 2020-09-17 09:09:52 -05:00
apkallum
594d9e49ce first attempt to migrate to Pathlib 2020-09-17 09:09:52 -05:00
Cristian Vargas
5e9b3099c6 Update fix_duplicate_links_in_index docstring
Co-authored-by: Nick Sweeting <git@sweeting.me>
2020-09-15 08:05:46 -05:00
Cristian
f55153eab3 feat: Update update command to work with querysets 2020-09-15 08:05:46 -05:00
Cristian
fe9604a772 feat: Add tests for remove command 2020-09-15 08:05:46 -05:00
Cristian
a8ed72501d feat: Refactor remove command to use querysets 2020-09-15 08:05:46 -05:00
Cristian
be520d137a feat: Refactor add method to use querysets 2020-09-15 08:05:46 -05:00
Cristian
be0dff8126 feat: Add tests to refactored init command 2020-09-15 08:05:46 -05:00
Cristian
404f333e17 feat: Refactor get_invalid_folders to work with a queryset instead of a list of links 2020-09-15 08:05:46 -05:00
Cristian
6b4b7127b4 feat: Remove unused imports 2020-09-15 08:05:46 -05:00
Cristian
b8585dd92e feat: load_main_index returns a queryset now 2020-09-15 08:05:46 -05:00
Cristian
c16fdf1b47 feat: Update data folder check 2020-09-15 08:05:46 -05:00
Cristian
874403e667 feat: Remove patch_main_index 2020-09-15 08:05:46 -05:00
Cristian
31343c1367 feat: Update extractors and add command to use sql index as source of truth 2020-09-15 08:05:46 -05:00
Cristian
02f36b2096 feat: Replace index.json with index.sql as the main index in init 2020-09-15 08:05:46 -05:00
Nick Sweeting
e87f1d57a3 fix linters 2020-08-18 09:22:12 -04:00
Nick Sweeting
f18d92570e wip attempt to fix timestamp unique constraint errors 2020-08-18 08:30:09 -04:00
Nick Sweeting
15efb2d5ed new generic_html parser for extracting hrefs 2020-08-18 08:29:05 -04:00
Nick Sweeting
5f84a7bc6e better handle the case where json index lags behind sql index 2020-08-18 08:13:13 -04:00
Nick Sweeting
77d2f08a5c show more info in merge conflict error message 2020-08-18 08:12:35 -04:00
Nick Sweeting
f371032b71 show warning when killing archivebox during index writing 2020-08-18 04:38:29 -04:00
Nick Sweeting
225b63b732 skip invalid urls at all stages 2020-08-17 03:12:17 -04:00
Cristian
c073ea141d feat: Initial oneshot command proposal 2020-07-29 11:19:06 -05:00
Cristian
6006b4f93b refactor: Organize code to remove flake8 issues 2020-07-24 12:25:25 -05:00
Cristian
100fa5d1f5 fix: Guess timestamps and add placeholders to support older indices 2020-07-24 09:24:52 -05:00
Nick Sweeting
02a2fefbba
Merge pull request #385 from apkallum/origin/output-permissions 2020-07-23 11:52:31 -04:00
apkallum
0ed2a23670 ensure correct permissions for output folder 2020-07-23 10:28:10 -04:00
Cristian
71f5f03a20 fix: Add notice for issues with index detail 2020-07-22 17:08:32 -05:00
Cristian
a5550b2105 fix: Rename logging folder to avoid naming conflicts (and circular import issues) 2020-07-22 11:02:13 -05:00
Cristian
f4d1b5121e refactor: Move logging.py to main module to avoid circular import issues 2020-07-17 18:00:04 -05:00