1467 Commits

Author SHA1 Message Date
8377dc63c0 Fix attribute error accessing response.text
The response object *should* be a requests.Response object, which *should*
have a .text property. However, there are situations where this is not the
case, and in those cases we now won't produce an AttributeError.
2017-09-15 10:06:06 +02:00
54bb506e10 Orphan finder: also interpret 24-char hex strings as ObjectIDs
This is necessary as some dynamic node properties have ObjectIDs saved
as strings.
2017-09-14 17:43:23 +02:00
d4facbf2e3 Orphan finder: store the orphan-files.txt file in STORAGE_DIR
This allows running the orphan finder inside a docker container.
2017-09-14 17:34:02 +02:00
ddc8fc0f5e Clarify celery beat schedule a bit 2017-09-14 17:15:11 +02:00
82d2921424 Added support for periodic Celery tasks.
You have to run "manage.py celery beat" for this to work too. Run
"manage.py celery beat -- --help" to get CLI option help.
2017-09-14 16:00:59 +02:00
5d137ac997 Added Celery task for refreshing file links
This includes a CLI command to kick off a single run for the Celery task.

This does *NOT* include a check to see whether the task is already running!
2017-09-14 15:12:25 +02:00
b06e17acf0 Added a little reminder about what to do when you add a Celery module 2017-09-14 15:10:54 +02:00
c272b5d4bb refresh_backend_links CLI: don't convert str → int multiple times 2017-09-14 15:10:32 +02:00
eba28b4eb4 File link refresh: report on every N refreshed links
This makes it easier to see what the Celery worker is actually working on
when refreshing a large number of links.

It'll report on every N refreshed links, where N = link_count/25 but
clamped to N ∈ [5, 100]
2017-09-14 15:10:09 +02:00
44f473221f File link refresh: ignore soft-deleted files 2017-09-14 15:06:37 +02:00
3be47056a0 Orphan finder: drop the per-project finding
Overall finding is much faster, at the expense of a bit more RAM.
2017-09-14 12:18:10 +02:00
be6746f7ab Fixed bug when parsing node without content type property 2017-09-14 12:09:54 +02:00
8cb506731f find_for_other: fail early when no project is given 2017-09-14 12:08:32 +02:00
e09649586f find_url_for_node: fail early when node has no valid project 2017-09-14 12:08:08 +02:00
230c15d51c Fix snag that happens when PUTting a user document without roles key. 2017-09-14 11:23:35 +02:00
c4a765e73b Change url of EXAMPLE_PROJECT
This prevents tests from breaking as we start implementing the concept of 'main' project.
2017-09-13 23:30:23 +02:00
18eb84fa9d Log capabilities at DEBUG level. 2017-09-13 16:36:36 +02:00
1a505bb0a2 Work around bugsnag issue
3263f0a551b663f93d1f2ec8087ef43302752f8b didn't fix it in production.
2017-09-13 16:36:29 +02:00
86caa3a044 Fixed inconsistency between requirements.txt and setup.py 2017-09-13 16:36:01 +02:00
3263f0a551 Upgraded bugsnag 2.3.1 → 3.1.1
I hope this fixes this error; I no longer see it locally:

Traceback (most recent call last):
  File "/data/git/blender-cloud/runserver.wsgi", line 16, in <module>
    application = PillarServer(my_path)
  File "/data/git/pillar/pillar/__init__.py", line 96, in __init__
    self._config_bugsnag()
  File "/data/git/pillar/pillar/__init__.py", line 191, in _config_bugsnag
    handle_exceptions(self)
  File "/opt/python/lib/python3.6/site-packages/bugsnag/flask/__init__.py", line 27, in handle_exceptions
    got_request_exception.connect(__log_exception, app)
  File "/opt/python/lib/python3.6/site-packages/blinker/base.py", line 130, in connect
    sender_ref = reference(sender, self._cleanup_sender)
  File "/opt/python/lib/python3.6/site-packages/blinker/_utilities.py", line 134, in reference
    weak = callable_reference(object, callback)
  File "/opt/python/lib/python3.6/site-packages/blinker/_utilities.py", line 145, in callable_reference
    return BoundMethodWeakref(target=object, on_delete=callback)
  File "/opt/python/lib/python3.6/site-packages/blinker/_saferef.py", line 135, in __new__
    key = cls.calculate_key(target)
  File "/opt/python/lib/python3.6/site-packages/blinker/_saferef.py", line 196, in calculate_key
    return (id(get_self(target)), id(get_func(target)))
  File "/opt/python/lib/python3.6/site-packages/events/events.py", line 41, in __getattr__
    (self.__class__.__name__, name))
AttributeError: type object 'PillarServer' has no attribute '__self__'
2017-09-13 16:03:38 +02:00
8aa6bb61dd Slightly nicer initialisation of Bugsnag 2017-09-13 16:02:48 +02:00
f89bec5089 Allow <pre> tags in comments
This is useful for code blocks.
2017-09-13 15:34:36 +02:00
896784a351 Clear session when token is invalid
Before this, the user's authentication token would still be stored in
the session even when it's found to be invalid. This caused a login
action to fail, but not in such a way that we would redirect to the login
page of Blender ID. Rather, it would keep you not logged in. By clearing
the session we're sure that the invalid token is forgotten, and the next
request will handle the login properly.
2017-09-13 15:23:38 +02:00
6488f4677e Be more graceful when URLer service isn't configured properly.
Errors are still logged, but find_url_for_node() will just act as if the
node doesn't exist when the URLer authentication token is invalid.
2017-09-13 15:22:04 +02:00
f650835c07 Orphan finder: soft-delete orphan files
This uses the orphan-files.txt file output by find_orphan_files() to
mark those files as deleted. This allows for a two-stage approach, where
file IDs are found on one machine (against a read-only MongoDB slave, for
example) and soft-deleted on another machine (against a writable master).
2017-09-13 14:05:28 +02:00
33feaa81ca Orphan finder: refuse to find orphans when orphan-files.txt exists. 2017-09-13 14:05:28 +02:00
16bf193b0e Added soft-delete to the files schema.
This allows us to soft-delete orphan files, at least until we know
that the orphan file detection is solid and can really be trusted.
2017-09-13 14:05:28 +02:00
ce59bc3335 Orphan finder: write file object IDs to orphan-files.txt 2017-09-13 14:05:28 +02:00
46d834f5aa Orphan finder: log duration of orphan file search 2017-09-13 14:05:28 +02:00
4be05c8b57 Orphan finder: when iterating all projects, gracefully stop at CTRL+C 2017-09-13 14:05:28 +02:00
5ce02bbbfe Orphan finder: fix bug when no orphan files are found 2017-09-13 14:05:28 +02:00
d01b498ad5 Orphan finder: Default project._deleted to False 2017-09-13 14:05:28 +02:00
b1d69b2304 Added orphan file finder. Works per project or pass 'all' for all projects.
This is quite a heavy thing to run, since it goes over all files of a
project, and then goes over every document in (almost) every collection
which has a property 'project' that's set to the project ID. It then goes
over every document to find all ObjectIDs and removes those from the set
of file ObjectIDs for that project. The remaining ObjectIDs are considered
orphans.

This is a very thorough search, but it doesn't require any knowledge of
the document and collection structure, so it should be future-proof.
2017-09-13 14:05:28 +02:00
9ac870e0a5 Fixed scrolling to comment when the comment ID is in the URL hash. 2017-09-13 14:05:18 +02:00
a8511c9db5 Gracefully handle read timeouts when communicating with BlenderID 2017-09-12 16:30:11 +02:00
ab7d623d27 Create some indices used for statistics 2017-09-12 11:58:31 +02:00
901fea3361 Do not assume that users and groups keys exist in permissions 2017-09-11 22:35:44 +02:00
Dalai Felinto
3329788be8 Localization: Fix setup.py install/develop
After I removed the @translation_manager.command decorators the setup
install was failing because translations was not found in pillar.cli.
2017-09-11 21:54:34 +02:00
Dalai Felinto
303a33c3bf Internationalization: Backend support to localization based on user browser
User experience
===============
For users it means we can provide localized web-sites to enrich their
overall experiences.

Although for the Blender Cloud this doesn't make much sense (since the
content is in English), Flamenco and Attract can really benefit from
this.

New configuration settings
==========================
There are two new parameters in config.py:

* DEFAULT_LOCALE='en_US'
* SUPPORT_ENGLISH=True

They are both properly documented in the `config.py` file.

Technicall details
==================
We are using the 'Accept-Languages' header to match the
available translations with the user supported languages.

If an extension has a `translations` folder, it's used for translations.
However the main application (e.g., Blender Cloud) is the one that
determines the supported languages based on its `languages` folder.

How to mark strings for translation
===================================
See the documentation in README.md.

But as an example, 404.pug and pillar/__init__.py::handle_sdk_resource_invalid
have marked up strings that will be extracted once you install pillar,
or run any of the translations commangs.

Remember to **gulp** after you update the template files.

How to setup translations
=========================
You will need to create translation for the main project, and for each
extension that you want to see translated. I added a new entry-point to
the installation of Pillar.

So all you need is to use the `translations`
script to initialize, update and compile your translations.

Pending tasks
=============
Aside from marking more strings for extraction and start the translation
effort it would be interesting to replace the pretty_date routine with
momentjs.

Acknowledgement
===============
Many thanks for Sybren Stüvel for the suggestions and throughout code
review. Thanks also to Francesco Siddi for the original documentation
and suggesting me to tackle this. And Kudos for Pablo Vazquez for the
motivational support and for the upcoming "strings mark up" task force!

The core of the implementation is based on Miguel Grinberg i18n chapter
of his great 'The Mega Flask Tutorial'.

Reviewers: sybren

Differential Revision: https://developer.blender.org/D2826
2017-09-09 00:26:18 +02:00
Dalai Felinto
b769cc6c3c Fix app_root in unittests 2017-09-08 14:06:55 +02:00
4e5ce71a52 File storage link refreshing: log nr of documents to refresh. 2017-09-07 15:53:16 +02:00
7dbd0c7896 Exclude Flamenco task logs when dumping the database. 2017-09-07 15:52:57 +02:00
Dalai Felinto
d5a55f71a7 Fix Flask and Eve dependency issue
Eve expects 'flask>=0.10.1,<=0.12' so we can't use flask==0.12.2 in
pillar for now.

This needs to be fixed upstream (eve), but for now `python setup.py
install`.
2017-09-06 17:51:21 +02:00
ebcd280886 Updated requirements in setup.py 2017-09-06 17:30:55 +02:00
Dalai Felinto
216b9278af A user should not be able to vote on own content
This should be hidden in the UI as well, but the backend should support this too.
We also want to set initial rating of 1 for contents that need it.

This commit includes a new unittest for this case.

Reviewers: sybren

Differential Revision: https://developer.blender.org/D2825
2017-09-06 13:51:32 +02:00
eb467474f8 Make our require_login() optionally redirect to the login page
This mimicks the behaviour of flask_login. In our case, it only redirects
when redirect_to_login=True and the user is anonymous. Otheriwse it still
results in a 403 Forbidden response.
2017-09-06 12:07:20 +02:00
e5b80297ba Fixed user ID, we now pass a UserClass instance to compute the capabilities 2017-09-05 16:05:57 +02:00
6b32c4ede8 Use has_cap('admin') instead of has_role('admin') 2017-09-05 16:05:39 +02:00
2db0ee11db Fixed case for packages.
'pip freeze -r requirements.txt' otherwise complains about this.
2017-09-05 13:38:04 +02:00
146bf97432 Removed doubly-listed CommonMark package 2017-09-05 13:37:43 +02:00