Commit Graph

236 Commits

Author SHA1 Message Date
Andrew Dolgov f4ae0f053b update: remove debugging block 2013-05-02 10:26:32 +04:00
Andrew Dolgov 566417c4e7 restore updstart threshold 2013-05-02 02:31:32 +04:00
Andrew Dolgov 5d3e5a1bb2 simplify feed cache age handling (reduce caching to sequential updates) 2013-05-02 02:30:53 +04:00
Andrew Dolgov 5de4010487 disable marking for the time being 2013-05-02 02:21:11 +04:00
Andrew Dolgov 5ef8409700 move the precautionary timestamp bumping 2013-05-02 02:20:34 +04:00
Andrew Dolgov 5d0d3887af add _DISABLE_HTTP_304 2013-05-02 02:11:11 +04:00
Andrew Dolgov 15c762beda updater: show owner_uid for checked feeds 2013-05-02 02:02:49 +04:00
Andrew Dolgov 52637d3b30 remove cache valid bailout clause 2013-05-02 01:36:17 +04:00
Andrew Dolgov 865a3ed6a0 change feed cache file extension 2013-05-02 01:33:02 +04:00
Andrew Dolgov d4992d6b48 add support for dc:subject and slash:comments 2013-05-01 20:55:08 +04:00
Andrew Dolgov ee78f81ccd update: better tag-related debugging info 2013-05-01 20:33:59 +04:00
Andrew Dolgov 852d4ac890 support RDF-XML feeds 2013-05-01 20:30:52 +04:00
Andrew Dolgov fd0daa9b55 remove simplepie 2013-05-01 19:14:48 +04:00
Andrew Dolgov 431e27851b actually save feed xml in the cache 2013-05-01 18:10:27 +04:00
Andrew Dolgov 99429e57e4 remove simplepie entity decode hacks 2013-05-01 18:07:05 +04:00
Andrew Dolgov b8f316dc28 change caching to save xml data, remove RDF init section 2013-05-01 17:56:21 +04:00
Andrew Dolgov 04d2f9c831 add basic rss support 2013-05-01 17:38:16 +04:00
Andrew Dolgov cd07592c29 add basic tinyparser/atom 2013-05-01 17:04:57 +04:00
Andrew Dolgov 65c8d5e76d update: set last_updated to now() when process starts 2013-05-01 15:43:56 +04:00
Andrew Dolgov 39ede9862f experimental: decode numerical utf entities on import in entry title 2013-04-29 16:59:36 +04:00
Andrew Dolgov efe46a3b53 Merge pull request #167 from KonishchevDmitry/pr-allow-slash-in-filter-regex
Allow slashes in filter regular expressions
2013-04-27 02:10:27 -07:00
Dmitry Konishchev 7b80b5e160 Match each tag separately against user filter regular expression
Each article's tag should be matched against user filter regular
expression separately. Current matching confuses when you want to match
an exact tag. You suppose to write "^tag$", bug now have to write
"(^|,)tag(,|$)" which is very inconvenient and requires knowledge about
how do you process this matching.
2013-04-26 16:30:25 +04:00
Dmitry Konishchev ffa1bd7b19 Allow slashes in filter regular expressions
User's regular expressions need escaping before passing them to
preg_match()
2013-04-26 15:46:48 +04:00
Andrew Dolgov 90e5f4f1de base if-modified-since on last received article, not feed update timestamp 2013-04-25 18:42:48 +04:00
Andrew Dolgov 23283f11a3 fetch: better checking for 1970- date 2013-04-25 16:12:49 +04:00
Andrew Dolgov aafd55bab6 do not needlessly mark icons as failed 2013-04-24 19:01:51 +04:00
Andrew Dolgov 84ceb6bd2e fix floicon failed message not respecting debugging setting 2013-04-24 16:48:47 +04:00
Andrew Dolgov e2cf81e214 unify houskeeping stuff, increase spawn interval 2013-04-24 14:54:59 +04:00
Andrew Dolgov 654650857f reduce daemon chattiness a bit 2013-04-24 14:26:05 +04:00
Andrew Dolgov 8cabc200d5 increase updstart locking interval to prevent concurrent updates; decrease batch size for the daemon 2013-04-24 14:23:38 +04:00
Andrew Dolgov fce451a4f7 fix updating for pgsql
increase default batch size
remove random check between feedbrowser or update task executing
2013-04-24 13:43:40 +04:00
Andrew Dolgov 81c2066345 updating: do not use cache timestamp if it is unusable for current feed;
sort chronologically instead of random
2013-04-24 13:21:40 +04:00
Andrew Dolgov 6ee0d4b05c do not recalculate avg color repeatedly 2013-04-23 22:37:06 +04:00
Andrew Dolgov 560cbd8c0d attempt to recalculate favicon avg color if it changes 2013-04-23 22:33:23 +04:00
Andrew Dolgov 36490f11a3 add a hack so that if floicon fails on some icon, it won't be processed again 2013-04-23 22:30:06 +04:00
Daniel Andersson 1134497129 Fix doubly applied basepaths to lockfiles. 2013-04-19 21:25:26 +02:00
Andrew Dolgov 68cccafcad remove debug_enabled global 2013-04-19 13:42:40 +04:00
Andrew Dolgov 9ec103525d update_rss_feed: simplify debug reporting 2013-04-19 13:26:10 +04:00
Andrew Dolgov ebec81a6fb subscribe: verify XML before adding to the database; fetch: try to work around entity problems if initial parsing fails 2013-04-19 13:17:28 +04:00
Andrew Dolgov 1367bc3f5e trim received feed data on fetch 2013-04-19 12:35:30 +04:00
Andrew Dolgov 96db9a14d9 do not try to calculate icon avg color if GD is not present 2013-04-19 08:42:35 +04:00
Andrew Dolgov ce2b219c64 entry_author: do not double escape string, author & comments: trim 2013-04-18 16:25:02 +04:00
Andrew Dolgov 1ffe3391f9 make pluginhost a singleton 2013-04-18 12:27:34 +04:00
Andrew Dolgov 52d88392da move db-prefs to OO 2013-04-18 12:00:01 +04:00
Andrew Dolgov 3f6f085769 add better gzdecode support for php native fetch, use curl under open_basedir 2013-04-18 11:36:13 +04:00
Andrew Dolgov d0d86316ee Revert "do automatic gzcdecode on feed data"
This reverts commit aea03558bb.
2013-04-18 11:19:25 +04:00
Andrew Dolgov aea03558bb do automatic gzcdecode on feed data 2013-04-18 09:15:55 +04:00
Andrew Dolgov 48cb253625 get_article_filter: fix missing argument 2013-04-17 19:25:15 +04:00
Andrew Dolgov a42c55f02b fix blank character after opening bracket in function calls 2013-04-17 18:34:18 +04:00
Andrew Dolgov 7440a7fe3c daemon2: use passthrough to daemon-loop 2013-04-17 18:19:28 +04:00
Andrew Dolgov 6322ac79a0 remove $link 2013-04-17 16:48:41 +04:00
Andrew Dolgov 857efe49e6 add schema checking to feed updating and update script 2013-04-17 12:10:35 +04:00
Andrew Dolgov e2261e177b implement old log entry purging in update_daemon_common; add some more error types; tweak error log display 2013-04-16 20:34:20 +04:00
Andrew Dolgov 5e3d5480f7 use hashed guids in a hopefully backwards compatible fashion 2013-04-15 19:28:11 +04:00
Andrew Dolgov bc7a144dd5 properly remove and replace favicon color when favicon gets manually removed/uploaded 2013-04-15 18:22:48 +04:00
Andrew Dolgov 12a6bd28ac remove hsl stuff from classes/feeds, we take care of this when choosing the color 2013-04-15 17:07:06 +04:00
Andrew Dolgov 6ac722d5b1 try to improve color guessing algorithm a bit 2013-04-15 17:01:51 +04:00
Andrew Dolgov 63c323f736 only enable colored favicons in vfeeds; fix RGB triplet packing 2013-04-15 15:17:12 +04:00
Andrew Dolgov 687a4f59c8 store favicon color as #rrggbb; process existing favicons 2013-04-15 15:08:00 +04:00
moontear 2cfbb448fb Added average color calculation of feeds' favicons for banded display. 2013-04-15 12:28:52 +02:00
Andrew Dolgov 044cff2d74 implement basic feed authentication parameter encryption in the database (FEED_CRYPT_KEY) 2013-04-13 18:24:41 +04:00
Andrew Dolgov c5569ff8d3 decode html entities in article title on import 2013-04-12 16:30:40 +04:00
Andrew Dolgov 3306daecf4 implement upload-related support for open_basedir 2013-04-11 19:12:00 +04:00
Andrew Dolgov 5929a0c1e4 updater: prevent duplicate feed ids being scheduled for update 2013-04-04 13:20:44 +04:00
Barak Korren 32b7a340ad Merge branch 'master' into tunable-fetches
Conflicts:
	include/rssfuncs.php
2013-04-01 16:05:48 +03:00
Barak Korren 8401101d50 Moved hard-coded fetch timeout values into config file to allow site-level tuning 2013-04-01 15:08:21 +03:00
Andrew Dolgov 5e736e4592 implement stop action 2013-03-31 19:28:36 +04:00
Andrew Dolgov ee0542ce28 daemon: do not schedule user-disabled feeds (2) 2013-03-31 14:36:06 +04:00
Andrew Dolgov cd7ebb39e4 daemon: do not schedule user-disabled feeds 2013-03-31 14:24:13 +04:00
Andrew Dolgov e81610d9ef daemon: sort by random 2013-03-30 16:28:21 +04:00
Andrew Dolgov 8292d05b7c daemon: show feeds/minute stats 2013-03-30 16:25:10 +04:00
Andrew Dolgov 17e74b21cf update_rss_feed: bailout if cache is considered valid, but older than last_updated 2013-03-30 15:42:27 +04:00
Andrew Dolgov 31623bfa99 if-modified-since: also take last_updated into account 2013-03-30 15:35:16 +04:00
Andrew Dolgov 7a01dc776e support If-Modified-Since 2013-03-30 15:12:20 +04:00
Andrew Dolgov 7b11aa80ab Revert "Avoid updating feed based on user pref DEFAULT_UPDATE_INTERVAL if it is set to -1 (no update)"
This reverts commit bca1cfbb95.
2013-03-30 13:26:15 +04:00
all bca1cfbb95 Avoid updating feed based on user pref DEFAULT_UPDATE_INTERVAL if it is set to -1 (no update) 2013-03-30 09:43:37 +01:00
Andrew Dolgov 0d6943b097 Revert "order feeds selected for update randomly"
This reverts commit 13b33a3fc3.
2013-03-29 17:25:07 +04:00
Andrew Dolgov 13b33a3fc3 order feeds selected for update randomly 2013-03-29 15:36:40 +04:00
Andrew Dolgov 32b05702ff daemon: do not show feed urls twice 2013-03-29 14:32:11 +04:00
Andrew Dolgov 30ac8d1f86 update: exclude feeds with disabled updates 2013-03-29 12:31:53 +04:00
Andrew Dolgov 1c4421fcd7 experimentally rework update daemon to process all feeds with the same url sequentially to minimize network access and better utilize caches 2013-03-29 12:21:40 +04:00
Andrew Dolgov 4f9cbdff1d cache simplepie object instead of xml feed data 2013-03-29 11:50:58 +04:00
Andrew Dolgov d5974daa33 remove feed update interval debugging 2013-03-29 11:32:22 +04:00
Andrew Dolgov f074ffe945 cache downloaded feed data to minimize network access 2013-03-29 11:31:10 +04:00
Joschasa ca5ff2d9fb Remove 2 useless line breaks. One got printed even with --quiet. 2013-03-28 20:30:06 +01:00
Andrew Dolgov be574731fc modify sorting by date_entered: set date_entered per-batch; use updated as a secondary criteria 2013-03-27 13:34:04 +04:00
saperduper 30123fe630 Handle future pubDate
Posts with pubDate in the future always appear at the top, which is ruining the
whole "newest at top" idea. One way to handle this is to use time() instead.
This is what google reader does and seems reasonable.
2013-03-25 18:49:11 +02:00
Andrew Dolgov a3a896a127 let us rejoice on being able to create double negative filters (refs #631) 2013-03-25 19:46:43 +04:00
Andrew Dolgov 227d5e4168 label auto matcher: preg_quote() label 2013-03-24 22:49:43 +04:00
Andrew Dolgov 9811276da7 rssfuncs: do not try to match on empty label 2013-03-23 09:23:02 +04:00
Andrew Dolgov 3972bf5981 db_escape_string: specify link parameter for consistency; sessions: do not force-close db connection in _close() 2013-03-22 09:14:55 +04:00
Andrew Dolgov f0bd8e6531 rework image caching to work without permanent article content rewriting (refs #582) 2013-03-19 09:25:36 +04:00
Andrew Dolgov f935d98e5d properly fix the cache_images issue (refs #578) 2013-03-19 00:42:59 +04:00
Andrew Dolgov 7873d58822 implement proper last_marked/last_published feeds for proper sorting of
published and marked virtual feeds, remove sorting by last_read
workaround
api: add pubsubhubbub ping when article is being set published
bump schema
2013-03-17 15:38:21 +04:00
Andrew Dolgov 1f45c8571f fix escaping of entry content being broken by cache_images() rewriting 2013-03-16 18:50:13 +04:00
Andrew Dolgov 8349a32e75 remove cache_content remnants 2013-03-16 16:11:58 +04:00
Andrew Dolgov cc38c8e549 remove LIBXML_NOEMPTYTAG because of double <br/>s - the #357 issue with
htmlpurifier might not be relevant anymore because of htmLawed switch,
but <br/>s are annoying
2013-03-16 16:07:11 +04:00
Andrew Dolgov 96f98cb091 escape fetch error message before saving in the db (closes #550) 2013-03-11 14:14:28 +04:00
Andrew Dolgov 41acdf4c11 remove purging of inactive feeds in daemon_common (causes duplicates) 2013-03-07 09:03:42 +04:00
Andrew Dolgov f161869541 when purging inactive feeds, set last_updated to NOW() 2013-03-04 21:49:14 +04:00