3534b8dfa7
improve logging for per-feed update task failures
2020-09-28 08:02:38 +03:00
74cd60d7cc
update_rss_feed: don't return as if failed on http 304
2020-09-27 17:13:36 +03:00
d4d0e976dc
update-feed: exit with non-zero exit code if update_rss_feed() failed
...
daemon: log if per-feed update task terminated with non-zero exit code
2020-09-27 16:42:45 +03:00
0761533d0a
lock per-feed update processes based on feed ID to reduce possibilty
...
of concurrent updates
2020-09-27 16:01:39 +03:00
528b387563
update individual feed in a separate process to prevent PHP fatal errors
...
(for example, OOM) from stopping the entire batch
this should also slightly increase memory budget for update processes
2020-09-27 15:58:13 +03:00
05744bb474
fix updater never scheduling feeds for update if they never been updated before while having default update interval set
2020-09-22 20:33:51 +03:00
6811d0bde2
use self:: in some places to invoke static methods from the same class
2020-09-22 14:54:15 +03:00
74568df4ff
remove a lot of stuff from global context (functions.php), add a few helper classes instead
2020-09-22 09:04:33 +03:00
3dd4169b5f
clarify some URL validation-related error messages
2020-09-21 20:35:24 +03:00
4785f21316
update_rss_feed: log effective URL after fetching
...
validate_url: treat scheme as case-insensitive
2020-09-21 20:26:57 +03:00
a4525d31b2
replace FALSE with false so that static analyzer shuts up about it
2020-09-17 19:02:27 +03:00
afa0023c51
don't try to update manually disabled feeds even if they haven't been updated before or are marked for a manual update
2020-09-17 15:40:50 +03:00
c352e872e9
core: pass found enclosures to HOOK_ARTICLE_FILTER
...
af_redditimgur: remove enclosures if we found something to embed because it's going to be a low-res thumbnail
2020-06-24 22:54:14 +03:00
6eb94f1e13
better support for image srcset attributes as discussed in https://community.tt-rss.org/t/problem-with-img-srcset/3519
2020-06-15 11:58:59 +03:00
06d2c65193
calculate_article_hash: don't die() on previous, woops
2020-05-17 17:44:32 +03:00
3a142cbf58
calculate_article_hash: ignore some useless or read-only fields (i.e. GUID) when calculating hash
2020-05-17 17:42:37 +03:00
cd1f3cb8cc
* store UID in article hashed GUID separately so it could be migrated cleanly to a different instance
...
* store resulting GUID as a JSON object so it could be extended easier if needed
2020-05-17 14:01:16 +03:00
3a4b9249a9
DiskCache: properly deal with srcset attributes
2020-04-29 19:29:36 +03:00
4a00f96733
remove unneeded var_dump()
2020-04-29 11:35:02 +03:00
6573541873
* add HOOK_ENCLOSURE_IMPORTED
...
* pass feed id to HOOK_FEED_PARSED
2020-04-29 11:33:39 +03:00
ec1b0befc7
add support for video[@src] in media cache
...
it's a valid alternative to a source[@src] child element:
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video
2020-03-12 11:08:39 +01:00
cdde23b4dc
actually download <video> posters to media cache
...
video[@poster] is already supported in the rewriting logic but never
actually downloaded
2020-03-12 11:08:33 +01:00
f24ece85a6
add validationtextarea control, use it for filter match editor
2020-02-28 13:53:45 +03:00
6080cca9ca
scrap counter cache system; rework counters to sum() booleans instead
2020-01-24 14:25:31 +03:00
e5b7b145e5
cache media: set referrer to source URL when fetching images
2019-11-25 09:48:24 +03:00
304d3a0b88
tag-related fixes
...
1. move tag sanitization to feedparser common item class
2. enforce length limit on tags when parsing
3. support multiple tags passed via one dc:subject and other such elements, parse them as a comma-separated list
4. sort resulting tag list to prevent different order between feed updates
5. remove some duplicate code related to tag validation
6. allow + symbol in tags
2019-11-20 18:56:34 +03:00
8c3efd51ec
reset domain hit quota on feed update start
2019-11-17 13:17:21 +03:00
0d7b10469b
update_rss_feed: add specific logging for HOOK_FETCH_FEED, HOOK_FEED_FETCHED, HOOK_FEED_PARSED handlers
2019-11-14 06:39:45 +03:00
5bb8dad631
is_gzipped: don't try to strpos() over entire buffer
2019-11-12 07:11:10 +03:00
647c7c45eb
allow article filters to modify num_comments
2019-10-25 14:37:00 +03:00
4e05008aac
update_rss_feed: force cast initial timestamp value to integer
2019-09-30 11:41:07 +03:00
b0d67cd3d0
rework previous to pass unformatted timestamp to plugin, and deal with formatting later
...
also, move timestamp-related debugging output after plugin handler
2019-09-11 14:04:59 +03:00
94a12b9674
pass formatted entry timestamp to article filters and allow them to modify it
2019-09-11 11:43:40 +03:00
6914ad1f74
retire MIN_CACHE_FILE_SIZE
2019-08-14 12:44:50 +03:00
84974c60a7
RSSUtils::cache_media, cache_enclosures: use DiskCache
2019-08-14 12:15:56 +03:00
fdb6066bf6
* HOOK_ENCLOSURE_ENTRY: pass article_id to handler
...
* DiskCache: multiple fixes; support isWritable() for cache entries, set content-disposition for send()
* public/cached_url: allow selecting files from sub-caches other than images
* plugins/Cache_Starred_Images: rework to use DiskCache, can be enabled per-user, properly handles article enclosures, etc
2019-08-13 16:40:21 +03:00
19b9b27662
expire_cached_files to DiskCache::expire()
2019-08-13 14:13:42 +03:00
088fcf8131
move more globals to more appropriate places
...
set libxml to always use internal errors
2019-06-20 08:40:02 +03:00
4fa9aee4e7
move several more global functions to more appropriate classes
2019-06-20 08:14:06 +03:00
9423d72f6c
parser: force libxml error messages to valid utf8
2019-05-12 10:13:22 +03:00
c936cc3a1f
use DEFAULT_SEARCH_LANGUAGE to generate tsvector index if per-feed language is not specified, also use it as default value on search form for convenience
2019-04-10 13:03:26 +03:00
671f4cee65
domdocument: remove old meta charset unicode hacks, replace with shorter xml preamble utf8 hack (on loadhtml where it makes sense)
...
af_readability: better (?) charset hack for non-unicode pages
2019-03-21 21:08:02 +03:00
33a2d5f8e4
update_rss_feed: set basic feed info if site_url is blank
2019-03-15 14:00:09 +03:00
69a691f4e1
cleanup old feed browser cache
2019-03-06 20:12:44 +03:00
0b74db5ad7
remove feedbrowser (other feeds)
2019-03-06 20:02:06 +03:00
38e01270d8
archived feeds: expire old entries (schema bump)
2019-03-06 19:06:05 +03:00
13e7e775a3
update_rss_feed: mark_unread_on_update should take into account catchup filter action and entry_force_catchup
2019-02-06 22:56:14 +03:00
949bfa3457
add minor clean()-ing on some rss feed values
2018-12-26 09:58:28 +03:00
eedd402807
rssutils: don't gzdecode() stuff
2018-12-21 17:52:41 +03:00
a5517fe857
fetch_file_contents: decompress gzipped data
...
af_readability: remove utf8 preamble hack
2018-12-21 17:50:16 +03:00
958fbfedb6
rssutils: check if returned data is in gzip format before trying to decode it
2018-12-14 14:55:36 +03:00
4b2f3039d2
Properly report filter plugin time (re-fixes PR 98).
2018-12-12 21:30:16 -05:00
53602096b9
Fixed misplaced bracket.
2018-12-12 11:47:36 -05:00
f3737c0b24
update_rss_feed: add log message if article is filtered out
...
combine filters: fix crash on missing global function
2018-12-08 17:01:30 +03:00
1e3a53c037
do not try to update filter triggers if nothing was triggered (properly this time)
2018-12-06 23:12:20 +03:00
5780a5d501
do not try to update filter triggers if nothing was triggered
2018-12-06 23:05:02 +03:00
3e4326e34d
add ttrss_filters2.last_triggered (bump schema version)
2018-12-06 19:37:20 +03:00
a01c33d654
add HOOK_FILTER_TRIGGERED (for filter debugging)
2018-12-06 19:15:00 +03:00
3ad9944d5e
fix missing sprintf() argument
2018-11-30 09:19:00 +03:00
c10a43069e
debug logging system rework:
...
* support various logging levels per-message
* remove hacks like debug_suppress, DAEMON_EXTENDED_DEBUG, etc
* _debug() is kept as a compatibility shim for plugins
2018-11-30 08:34:29 +03:00
2d54eb1a87
remove cache/simplepie
2018-11-23 12:31:47 +03:00
2c940c4861
better handle PDOExceptions during open transaction in feed update
2018-11-03 15:00:58 +03:00
665495b94b
cache_media: only touch() local file if it's writable
2018-09-10 16:17:12 +03:00
62d0060aa1
update_daemon_common: do not abort entire batch if PDOException happens when processing individual feeds
2018-09-03 13:03:35 +03:00
8ab77d19ef
Merge branch 'pullreq-enclosure-content-type' of tkappe/tt-rss into master
2018-08-21 03:47:39 +00:00
ac8a0e7dc6
Differentiate enclosures based on content type.
...
Some RSS feeds contain multiple enclosures with the same URL. When the first of
these is not recognized as an image, later entries are not added to the
database as rows in ttrss_enclosures. This change differentiates enclosures
based on their content type, so an entry can have multiple enclosure types with
the same URL (but possibly a different content type).
2018-08-18 17:17:30 +01:00
163b50b15f
cache_media: only show downloading debug message when actually downloading
2018-08-16 11:56:51 +03:00
069aea5989
remove FEED_CRYPT_KEY and everything related to it
...
always assume auth_pass_encrypted is false
2018-08-13 15:59:24 +03:00
3bbaf902ab
Sanitize language obtained for an entry.
2018-08-12 16:12:34 +01:00
22a866edb5
Store language of entries as indicated by the feed.
2018-08-12 15:27:26 +01:00
2b8afd4942
Only strip utf8mb4 if mysql_charset != utf8mb4
...
If a user has fixed their database properly utf8mb4 works just fine allowing emoji and other 4 byte unicode characters to work.
2018-08-04 13:07:32 +00:00
6e6c3a878d
update_rss_feed: limit maximum length of tsvector data because of pgsql limitations
2018-07-10 14:08:55 +03:00
66fe33e769
bump date_updated when updated article data is saved to exclude it from purging (because it is still present in the originating feed)
2018-05-20 10:41:08 +03:00
963c22646b
pass tsvector data as a named parameter on article update, remove escaping hacks
2018-04-03 13:57:27 +03:00
5edf4b73a4
add a workaround to support numeric tags
2018-02-11 10:24:24 +03:00
7f4a404566
include: convert some spaces to tabs
2018-01-30 10:44:31 +03:00
102a01354b
strip utf8mb4 characters in enclosures on mysql
2018-01-29 23:24:11 +03:00
26ad257de5
Fixed time stamping of new unmarked/unpublished articles
2018-01-22 22:41:24 +00:00
d4c05d0be2
update_rss_feed: don't try to use quoted NOW() in query
2017-12-18 18:27:23 +03:00
aa16334f1f
Include NOW() in prepared SQL for rssutils.php
2017-12-10 23:05:52 +00:00
e6532439d6
force strip_tags() on all user input unless explicitly allowed
2017-12-03 23:35:38 +03:00
7c6f7bb0aa
fix some minor issues found by code analyzer
2017-12-03 23:08:04 +03:00
342e8a9eeb
move feeds cache directory to cache/feeds
2017-12-03 21:01:35 +03:00
93e70e36c2
force article content/etc to string when updating to avoid failing null constraint check
2017-12-03 13:32:24 +03:00
49a888ecce
rssutils: forbid question marks in tsvector data, PDO gets confused sometimes even by quoted ?s
2017-12-03 10:10:01 +03:00
187abfe732
main classes: remove sql_bool_to_bool() kludge
2017-12-03 09:35:59 +03:00
0500e14cc2
update_rss_feed: transaction lock article processing
2017-12-02 11:16:33 +03:00
0567016b40
rssutils: PDO
2017-12-02 09:56:34 +03:00
afcb105f4e
rssutils: start PDO switch
2017-12-02 08:38:57 +03:00
e50c8eaa4e
enforce unconditional requests every 6 hours even if server claims data is not modified
2017-11-30 13:12:28 +03:00
9d930af9e1
fetch_file_contents: improve error handling
...
1. if request fails get error string from http response status line
2. do not override http error with possible CURL/php specific last error
3. fix silent php error generated while processing response headers to get last modified value
2017-10-30 13:13:10 +03:00
f9ad33c2d8
allows favicons to be in Windows PC BMP format
2017-10-09 22:58:00 +02:00
3476690cbf
Only require an array of basic info from 'HOOK_FEED_BASIC_INFO'.
...
Removes the need for the plugin to provide feed content.
Gives plugins a chance to provide 'title' and 'site_url' basic info.
Falls back to attempting retrieval+parsing of the fetch URL if needed.
2017-09-26 20:42:33 -05:00
bec5ba93e2
Add 'HOOK_FEED_BASIC_INFO' to enable plugins to provide basic feed info.
...
It's expected the plugin will return content parsable by FeedParser, which
will act as an interface to the basic feed info. In the case of a plugin
that also uses 'HOOK_FETCH_FEED', both might return the same content.
The hook signature was made somewhat similar to 'HOOK_FETCH_FEED'.
2017-09-24 19:37:49 -05:00
153cb6d305
add support for http 304 not modified (no timestamp calculation bullshit like last time)
2017-08-17 14:40:21 +03:00
20d2195f13
rssutils: include comment count when calculating article hash
2017-08-06 20:20:39 +03:00
02f3992a5a
Revert "Revert "filters: support matching on multiple feeds/categories""
...
This reverts commit f5d174bda9
2017-07-02 20:37:52 +03:00
f5d174bda9
Revert "filters: support matching on multiple feeds/categories"
...
This reverts commit 0bf7e007bb
2017-07-02 20:37:01 +03:00
0bf7e007bb
filters: support matching on multiple feeds/categories
...
opml: update filter export/import for new format
2017-07-02 20:27:21 +03:00
93af11cb7a
update_daemon_common: do not escape feed_url twice, remove some comments and stuff
2017-06-13 20:09:33 +03:00
Andrew Dolgov
lllusion3418
JustAMacUser
fox
Tobias Kappé
BtbN
jsoares
Andrew Dolgov
Richard Mortimer
Andrew Dolgov
Gilles Grandou
wn_