244 Commits

Author SHA1 Message Date
2990738b5d Lazy Home: Lazy load latest blog posts and assets and group by week and
project.

Javascript tutti.js and timeline.js is needed, and then the following to
init the timeline:

$('.timeline')
    .timeline({
        url: '/api/timeline'
    });

# Javascript Notes:
## ES6 transpile:
* Files in src/scripts/js/es6/common will be transpiled from
modern es6 js to old es5 js, and then added to tutti.js
* Files in src/scripts/js/es6/individual will be transpiled from
modern es6 js to old es5 js to individual module files
## JS Testing
* Added the Jest test framework to write javascript tests.
* `npm test` will run all the javascript tests

Thanks to Sybren for reviewing
2018-11-12 12:57:25 +01:00
b4ee5b59bd Sync Blender ID badge as soon as user logs in
This adds a new Blinker signal `user_logged_in` that is only sent when
the user logs in via the web interface (and not on every token
authentication and every API call).
2018-10-10 16:54:58 +02:00
879bcffc2b Asset list item: Don't show user.full_name in latest and random assets 2018-10-04 12:30:05 +02:00
6ad12d0098 Video Duration: The duration of a video is now shown on thumbnails and bellow the video player
Asset nodes now have a new field called "properties.duration_seconds". This holds a copy of the duration stored on the referenced video file and stays in sync using eve hooks.

To migrate existing duration times from files to nodes you need to run the following:
./manage.py maintenance reconcile_node_video_duration -ag

There are 2 more maintenance commands to be used to determine if there are any missing durations in either files or nodes:
find_video_files_without_duration
find_video_nodes_without_duration

FFProbe is now used to detect what duration a video file has.

Reviewed by Sybren.
2018-10-03 18:30:40 +02:00
199f37c5d7 Tagged Asset: Added metadata
Video duration, Project link and pretty date
2018-09-26 11:29:15 +02:00
e9d247fe97 Added assertion in test to verify that the asset was deleted 2018-09-21 14:24:37 +02:00
1ddd8525c7 Remove references to node from projects when the node is deleted.
Removes node references  in project fields header_node, nodes_blog, nodes_featured, nodes_latest.
2018-09-21 14:23:47 +02:00
9a0da126e6 Fix failing tests
Failure was due to a new ‘slug’ key in the link dict.
2018-09-18 15:14:27 +02:00
8753a12dee Tweak unit test to support new embed code 2018-09-16 22:04:22 +02:00
f8d992400e Extend attachment shortcode rendering
The previous implementation only supported rendering
attachments within the context of a node or project document.
Now it also supports node.properties. This is a temporary
solution, as noted in the TODO comments.
2018-09-15 19:01:58 +02:00
263d68071e Add view_progress to nodes of type asset 2018-09-15 17:59:30 +02:00
9a9d15ce47 Generate project_navigation_links
This function generates a list of selected links for important nodes such
as Pages and Blog. This list of links is used in the templates to provide
high level navigation of a Project.
2018-09-13 16:35:53 +02:00
e8fb77c39b Badge sync: also support removal of all badges
Removal is stored as '' for the HTML. This way there is still the expiry
date, which means we won't repeatedly check for changes.
2018-09-12 15:29:45 +02:00
9a9ca1bf8b Synchronise badges with Blender ID
Synchronisation is performed in the background by the Celery Beat, every
10 minutes. It has a time limit of 9 minutes to prevent multiple refresh
tasks from running at the same time.

Synchronisation is also possible with the `manage.py badges sync` CLI
command, which can sync either a single user or all users.
2018-09-12 15:02:19 +02:00
0983474e76 Store Blender ID OAuth scopes in MongoDB + request badge scope too
This also changes the way we treat Blender ID tokens. Before, the Blender ID
token was discarded and a random token was generated & stored. Now the
actual Blender ID token is stored.

The Facebook and Google OAuth code still uses the old approach of generating
a new token. Not sure what the added value is, though, because once the
Django session is gone there is nothing left to authenticate the user and
thus the random token is useless anyway.
2018-09-12 15:02:19 +02:00
1401a6168f Always use urljoin to construct Blender ID URLs 2018-09-12 15:02:19 +02:00
85eab0c6cb No longer hash auth tokens + store the token scopes
This partially reverts commit c57aefd48b10ca3cabc9df162bc32efa62a6a21e.
The code to check against hashed tokens remains, because existing tokens
should still work.

The unhashed tokens are necessary for fetching badges from Blender ID.
2018-09-12 15:02:19 +02:00
e19dd27099 API endpoint /api/nodes/tagged/<tag>
This endpoint returns nodes in public projects that have the given tag.
The returned JSON is cached for 5 minutes.
2018-09-06 15:42:50 +02:00
e4fa32b8e4 Fixed bug in attachment code 2018-09-06 13:36:01 +02:00
0fcafddbd1 Added unit test for creating comments
We had an issue creating comments, so I wrote a test for it. The test
succeeds on a new project, so the problem lies with the older projects.
In the end it was the comment node type that still had
`{'coerce': 'markdown'}`.
2018-09-05 14:54:08 +02:00
2698be3e12 Saving & restoring video watching progress
Video progress updates:

- Mark as 'done' when 90% or more is watched.
- Keep 'done' flag when re-watching.

The video progress is stored on three events, whichever comes first:

- Every 30 seconds of video.
- Every 10% of the video.
- Every pause/stop/navigation to another page.
- When we detect the video is looping.
2018-09-04 12:16:24 +02:00
2ad5b20880 Quick hack to get /p/{url}/jstree working again
Apparently Eve is now stricter in checking against MONGO_QUERY_BLACKLIST,
and blocks our use of $regex when getting child nodes. See
`jstree.py::jstree_get_children()`
2018-08-30 13:59:23 +02:00
e8123b7839 Apparently the test client now uses `https://localhost.local/' as URL
Previously this was 'http://localhost/'
2018-08-29 11:27:00 +02:00
6d6a40b8c0 Empty lists don't seem to be stored in MongoDB any more
It looks like with the new Eve (or one of its dependencies) empty lists
aren't stored any more; rather than storing `{'org_roles': []}`, it skips
the `'org_roles'` key altogether. Not sure what caused this, as it was
mentioned in neither the Eve nor the PyMongo changelog.
2018-08-29 11:26:19 +02:00
a8a7166e78 Use self.assertRaises as context manager 2018-08-28 17:45:58 +02:00
8a0f582a80 Removed dependency on flask_pymongo 2018-07-13 17:08:06 +02:00
0fdcbc3947 Restored MarkDown conversion using 'validator': 'markdown' 2018-07-13 17:02:38 +02:00
8dc3296bd5 Schema change for IP range, use validator instead of type
Custom types became rather useless in Cerberus 1.0 since the type checker
is cripled (doesn't know field name, cannot return useful/detailed error
messages). Instead we use a validator now.
2018-07-13 15:03:35 +02:00
a699138fd6 Merge branch 'master' into wip-flask-one 2018-07-13 13:50:24 +02:00
466adabbb0 Added unit tests for IP range validation 2018-07-13 13:50:01 +02:00
aaf452e18b Fixed Cerberus canary unit test
Apparently it's no longer possible for Cerberus to validate its own schemas.
2018-07-13 12:02:40 +02:00
baa77a7de5 Merge branch 'master' into wip-flask-one 2018-07-13 11:43:57 +02:00
5fb40eb32b Simple unittests for Cerberus validation 2018-07-13 11:42:31 +02:00
549cf0a3e8 WIP on libraries upgrade 2018-07-12 15:23:57 +02:00
9f380751f5 Support for capabilities check in any shortcode
Use the @capcheck decorator on any shortcode that should support
this. Currently used by iframe and youtube.
2018-07-11 12:32:00 +02:00
9ee7b742ab Make more consistent use of BLENDER_ID_ENDPOINT
Now BLENDER_ID_ENDPOINT is used for the Blender ID OAuth config,
and it's directly accessed when building requests for Blender ID token
validation (without using utility functions).
2018-06-22 19:38:27 +02:00
58c33074c3 Fix unittest for jinja.do_markdown
We were passing invalid html to do_markdown, which was returning a valid
version, by closing the <script> tag.
2018-06-22 17:10:38 +02:00
0e14bdd09f Introduce rating functions
These hotness and confidence calculation algorithms come from Reddit
and have been tweaked based on our experience on the Dillo project.
2018-06-03 02:09:20 +02:00
c9789f46db {iframe} shortcode no longer requires cap=xxx 2018-04-04 15:44:52 +02:00
3e7722a567 Expand image for {attachment slug link=self}
Clicking on the image will no longer open it directly, but expand it
instead.
2018-04-03 15:44:24 +02:00
1ba1da49c3 Pass positional arguments to attachment render functions
This allows handling `{attachment slug link}` as a synonym for
`{attachment slug link=self}`.
2018-04-03 15:42:47 +02:00
67e8e7c082 Disallow spaces in attachment slugs
Slugs shouldn't have spaces. It also interferes with using slugs in
shortcodes.
2018-04-03 13:59:31 +02:00
cbb5d546ef Fixed CLI cmd upgrade_attachment_schema
It didn't add the {'coerce': 'markdown'}, which caused the
upgrade_attachment_usage CLI command to skip 'upgraded' nodes.
2018-04-03 12:49:34 +02:00
3b452d14ce Render attachments with shortcodes rather than slugs
The attachments should now be rendered using `{attachment slug}` instead
of `@[slug]`. The `link` attribute can be specified in the shortcode
(for attachments that support it), rather than in the attachment itself.

The attachment subdocument is now reduced to `{oid: File ObjectID}`, and
nodes without attachments should NOT have an `attachment` property at
all (previously it would be an empty dict). This makes querying for
nodes with/out attachments easier.

The CLI command `upgrade_attachment_schema` can do dry-run and remove
empty attachments:

- Added --go to actually perform the database changes.
- Remove empty attachments, so that a node either has one or more
  attachments or no attachments sub-document at all.

The CLI command `upgrade_attachment_usage` converts `@[slug]` to
`{attachment slug}`. It also takes into account 'link' and 'link_custom'
fields on the attachment. After conversion those fields are removed from
the attachment itself.

Simplified maintentance CLI commands that iterate over all projects:
I've moved the common approach (either run on one project or all of
them, skipping deleted ones, giving a message upon dry-run, and showing
duration of the command) to a new _db_projects() function. The new
function is now used by two recently-touched CLI commands; more of them
could be migrated to use this.
2018-04-03 10:59:20 +02:00
f4e0b9185b Shortcodes for YouTube and iframes
Added shortcodes 2.5.0 as dependency; Earlier versions corrupted
non-ASCII characters, see
https://github.com/dmulholland/shortcodes/issues/6

The rendered elements have a `shortcode` CSS class.

The YouTube shortcode supports various ways to refer to a video:

    - `{youtube VideoID}`
    - `{youtube youtube.com or youtu.be URL}`

URLs containing an '=' should be quoted, or otherwise the shortcodes
library will parse it as "key=value" pair.

The IFrame shortcode supports the `cap` and `nocap` attributes. `cap`
indicates the required capability the user should have in order to
render the tag. If `nocap` is given, its contents are shown as a message
to users who do not have this tag; without it, the iframe is silently
hidden.

`{iframe src='https://source' cap='subscriber' nocap='Subscribe to view'}`

Merged test code + added HTML class for shortcode iframes
2018-04-03 10:49:00 +02:00
fe7e078f8b Added unit test
Should have been part of prev commit.
2018-03-28 12:42:36 +02:00
f22dc4d92a Fixed PATCHing comments 2018-03-27 17:42:29 +02:00
218c3f0dca Fixed comment rendering 2018-03-27 17:13:12 +02:00
dfaac59e20 Cache Markdown'ed HTML in database
This is done via coercion rules. To cache the field 'content' in the
database, include this in your Eve schema:

    {'content': {'type': 'string', 'coerce': 'markdown'},
     '_content_html': {'type': 'string'}}

The `_content_html` field will be filled automatically when saving the
document via Eve.

To display the cached HTML, and fall back to display-time rendering if it
is not there, use `{{ document | markdowned('content') }}` in your template.

Still needs unit testing, a CLI command for regenerating the caches, and
a CLI command for migrating the node type definitions in existing projects.
2018-03-27 16:34:32 +02:00
d2a0a5ae26 Added CLI command 'maintenance purge_home_projects'
This command soft-deletes home projects when their owning user is no longer
there.
2018-03-27 15:45:32 +02:00