Commit graph

287 commits

Author SHA1 Message Date
rr-
06ab98fa70 server/search: fix sort:random breaking tags
Using sqlalchemy's subqueryload to fetch tags works like this:

1. Get basic info about posts with query X
2. Copy query X
3. SELECT all tags WHERE post_id IN (SELECT post_ids FROM query X)
4. Associate the resulting tags with the posts

When original query contains .order_by(func.random()), it looks like
this:

1. SELECT post.* FROM post ORDER BY random() LIMIT 10
2. Copy "ORDER BY random() LIMIT 10"
3. SELECT tag.* FROM tag WHERE tag.post_id IN (
       SELECT id FROM post ORDER BY random() LIMIT 10)
4. Disaster! Each post now has completely arbitrary tags!

To circumvent this, we replace eager loading with lazy loading. This
generates one extra query for each result row, but it has no chance of
producing such anomalies. This behavior is activated only for
queries containing "sort:random" and derivatives so it shouldn't hit
performance too much.
2016-08-27 01:21:59 +02:00
rr-
f8e91a10e8 server/search: refactor query factories 2016-08-27 01:19:29 +02:00
rr-
6d26b5c37a server/search: fix sort:random 2016-08-26 23:27:33 +02:00
rr-
fa60b42f65 server/search: improve post list performance 2016-08-26 17:57:20 +02:00
rr-
422b99ac8d server/search: add content-checksum 2016-08-26 16:26:06 +02:00
rr-
ffb87f1650 server/posts: defer flush; save content lazily
Rather than flushing the post right away only to find out that there
were validation errors, try to postpone flushing for as long as
possible.

The previous behavior has led to too eager spending of post IDs - each
flush calls nextval(post_id_seq), and postgres sequences are not
affected by transaction rollbacks, so each erroneous post creation
discarded a post ID, which has led to gaps in post IDs.
2016-08-26 15:09:08 +02:00
rr-
bb369efa99 server/general: disable autoflush 2016-08-26 14:41:05 +02:00
rr-
280a55046a server/db: make query counter thread-local 2016-08-24 12:31:55 +02:00
rr-
73a8542220 server/posts: make anon snapshots for anon uploads 2016-08-22 20:07:39 +02:00
rr-
61d084cc66 server/search: support 'submit:' for anon uploads 2016-08-22 19:45:25 +02:00
rr-
b7e9cbd541 server/posts: allow tagless posts 2016-08-21 23:40:01 +02:00
rr-
28bcbd33b9 server/posts: use SHA1 checksums
This changes the checksums to ones that are compatible with 1.x, which
relieves the migration script from recalculating the checksums for all
the posts.
2016-08-20 13:06:19 +02:00
rr-
80af79779d server/snapshots: rewrite 2016-08-16 21:51:25 +02:00
rr-
65119d69ab server/rest: urldecode incoming paths 2016-08-16 21:42:11 +02:00
rr-
0c2efc0be5 server/rest: work around Python bug 27777
(See http://bugs.python.org/issue27777)
2016-08-16 21:42:11 +02:00
rr-
e89a086d58 server/rest: fix reporting parameter type errors 2016-08-16 21:42:11 +02:00
rr-
ff7bbbdd8a server/db: fix upgrade (regression from 9aea55e) 2016-08-16 21:42:11 +02:00
rr-
6c29377f6b server/db: allow full DSN; use memdb in tests
The earlier commit is still relevant as it allows to integrate real
database when needed.
2016-08-16 21:42:09 +02:00
rr-
e688f39887 server/db: specify nullable for foreign keys
This is mostly stylistic change, but it clearly shows the intended
behavior should we detect more integrity violation errors.
2016-08-16 17:22:33 +02:00
rr-
522886ae6c server/db: fix user cascade deletions 2016-08-16 17:22:33 +02:00
rr-
a224297c4f server/db: poor fix for tag aliases' order
It's a hack, but it seems to work okay.
2016-08-16 17:22:33 +02:00
rr-
48af5160df server/search: fix negative offsets causing ISE 2016-08-16 17:22:33 +02:00
rr-
87b1ee4564 server/tests: use real database
I'm experimenting with snapshots and found following limitation of
SQLite: https://www.sqlite.org/isolation.html
2016-08-16 17:22:33 +02:00
rr-
0320a0b55b server/general: improve versioning effectiveness
...by integrating it with sqlalchemy that adds WHERE conditions for each
UPDATE and DELETE statement.
2016-08-16 17:22:33 +02:00
rr-
ef4af697c4 server/tags: fix tag sorting
Brainfart from d6942121e5
2016-08-14 17:54:15 +02:00
rr-
663aacdf82 server/tools: add lint script
Integrated both pylint and pycodestyle.
2016-08-14 16:46:50 +02:00
rr-
9aea55e3d1 server/general: embrace most of PEP8
Ignored only the rules about continuing / hanging indentation.

Also, added __init__.py to tests so that pylint discovers them. (I don't
buy pytest's BS about installing your package.)
2016-08-14 16:44:03 +02:00
rr-
af62f8c45a server/general: ditch falcon for in-house WSGI app
For quite some time, I hated Falcon's class maps approach that caused
more chaos than good for Szurubooru. I've taken a look at the other
frameworks (hug, flask, etc) again, but they all looked too
bloated/over-engineered. I decided to just talk to WSGI myself.

Regex-based routing may not be the fastest in the world, but I'm fine
with response time of 10 ms for cached /posts.
2016-08-14 16:43:35 +02:00
rr-
d102c9bdba server/tests: update func.posts tests 2016-08-14 16:43:35 +02:00
rr-
264f9ee70b server/tests: update func.mime tests 2016-08-14 16:43:35 +02:00
rr-
c23c401c4d server/tests: add func.tags tests 2016-08-14 16:43:35 +02:00
rr-
53e96ba41f server/tests: add func.tag_categories tests 2016-08-14 16:43:35 +02:00
rr-
81dfbaec98 server/tests: add func.users tests 2016-08-14 16:43:35 +02:00
rr-
03c74cb5a3 server/tests: add func.comments tests 2016-08-14 16:43:35 +02:00
rr-
65efc309a8 server/comments: catch bad IDs 2016-08-14 16:43:31 +02:00
rr-
56b3eb9674 server/util: fix case conversion 2016-08-14 16:43:04 +02:00
rr-
86452019a3 server/util: improve catching bad field names
KeyError could catch exceptions that happened inside the serializer
routine and mistakenly report them as an error with user input.
2016-08-14 16:43:04 +02:00
rr-
f6f07a35df server/general: authenticated_user->auth_user 2016-08-14 16:43:04 +02:00
rr-
c2bbf7b62c server/general: add assertions 2016-08-14 16:43:04 +02:00
rr-
bb86e9bf56 server/posts: add more safety checks for notes 2016-08-14 16:43:04 +02:00
rr-
7cd4a1a530 server/tags: verify description size 2016-08-14 16:43:01 +02:00
rr-
3db4f39545 server/tag-categories: correct exception type 2016-08-14 12:31:46 +02:00
rr-
92075bb455 server/tags: don't verify tag names while getting 2016-08-14 12:31:46 +02:00
rr-
d6942121e5 server/tags: change tags sort order 2016-08-14 12:31:21 +02:00
rr-
07237bc2bc server/tags: fix changing name case 2016-08-14 11:39:04 +02:00
rr-
7e5deee76b server/tags: fix tag merging 2016-08-14 11:39:04 +02:00
rr-
2b3d193b7c server/tags: don't auto-create tag categories 2016-08-14 11:38:59 +02:00
rr-
8d04df38fd server/general: add entity versions 2016-08-07 09:55:51 +02:00
rr-
9e2dace73f server/posts: improve errors for bad note points 2016-08-05 23:02:34 +02:00
rr-
5092c2c587 server/posts: respect tag creating privilege 2016-08-02 12:44:38 +02:00