From b64d4f222d4ca6058203b30b026aa4226fe58c34 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Demette?= Date: Mon, 15 Jun 2015 22:22:38 +0200 Subject: [PATCH] Complementary documentations --- Documentation/Database.md | 116 +++++++++++++++++++++++++++ Documentation/Design_pattern.md | 26 ++++++ Documentation/Ergonomics.md | 18 +++++ Documentation/Install.md | 83 +++++++++++++++++++ Documentation/Suggested_algorithm.md | 16 ++++ README.md | 7 ++ 6 files changed, 266 insertions(+) create mode 100644 Documentation/Database.md create mode 100644 Documentation/Design_pattern.md create mode 100644 Documentation/Ergonomics.md create mode 100644 Documentation/Install.md create mode 100644 Documentation/Suggested_algorithm.md diff --git a/Documentation/Database.md b/Documentation/Database.md new file mode 100644 index 0000000..7a0820d --- /dev/null +++ b/Documentation/Database.md @@ -0,0 +1,116 @@ +Database structure and optimization +=================================== + +## Which Database Engine ? + +Mewpipe must be ready to scale quickly and with ease. With that in mind, we had to make the choice of NoSQL. +MongoDB is an open-source document database engine, and the leading NoSQL database. +It's a good choice because this engine is agile and scalable through many ways: Replica sets and Sharding. + +## Our schema + +MongoDB permits evolution of our schema over time. Some keys must be omitted when empty. + +Our collections: + +* media +* media.chunks +* media.files +* media.shareCounts +* media.thumbnails.chunks +* media.thumbnails.files +* media.views +* system.indexes +* users + +### Users + +```JavaScript +{ + "_id" : ObjectId("555a076a2fd06c1891000001"), + "createdAt" : ISODate("2015-06-13T14:46:40.556Z"), //Bad time, fault on my old + "name" : { + "firstname" : "Foo", + "lastname" : "Bar", + "nickname" : "FooBar" + }, + "email" : "foo@bar.com", + "roles" : [ ], //List of roles (strings), available : admin + "hashedpassword" : "$2a$10$LD9dA75m2U6jx8cRRKmc.u5DQYMHACKym.4OZkJ0T91qSVJCZHvU2", // Bcrypt encoded + "usertokens" : [ + { + "token" : ObjectId("557cab510000000000000000"), + "expireat" : ISODate("2015-06-13T23:14:41.330Z") + } + ], + "twitter" : { + "userId" : "99999999" + } +} +``` + +HashedPassword uses [bcrypt](http://codahale.com/how-to-safely-store-a-password/). +When User tokens are cleaned, all older tokens are removed + +### Media + +When we store a media, it's splitted between many collections : + +* media, which store some metadata like title, summary, etc... And do the link with other media metadata +* media.files and media.chunks, which store binary (chunks) and metadata of the file (md5, content type...) +through GridFS +* media.thumbnails.files and media.thumbnails.chunks, which store binary (chunks) and metadata of the thumbnail +(md5, content type...) through GridFS + +```JavaScript +{ + "_id" : ObjectId("557c42502fd06c127000001c"), + "createdAt" : ISODate("2015-06-13T14:46:40.556Z"), + "title" : "Penguins_of_Madagascar.webm", + "summary" : "My amazing video", + "publisher" : { //Embed the user who publish the video + "_id" : ObjectId("555a076a2fd06c1891000002"), + "name" : { + "firstname" : "Admin", + "lastname" : "Admin", + "nickname" : "Admin" + }, + "email" : "admin@admin.com" + }, + "file" : ObjectId("557c42502fd06c127000001d"), //ObjectID of the file + "thumbnail" : ObjectId("557c42502fd06c1270000023"), //ObjectID of the thumbnail + "scope" : "public", + "views" : 2, //Cache of the number of views, calculated from media.views + "shares" : 2 //Cache of the number of shares, calculated from media.shareCounts, +} +``` + +### Shares & Views + +This collection groups views by user and media, the aim is to be able to calculate some statistics, +like the most viewed media of a user. + +```JavaScript +{ + "_id" : ObjectId("556e3ba94d96f23c00b302be"), + "media" : ObjectId("556dda352fd06c0fb6000002"), + "user" : ObjectId("555a076a2fd06c1891000001"), + "count" : 4 +} +``` + +An aggregation query calculates the number of views mongo-side and cache the result in the media. + +## Our strategy + +* Media and thumbnails are split in files of 16Mb, they are concatenated on the fly by MongoS (MongoDB Shard). +This is a powerful and fast system. +* The user is embedded inside the media. This method facilitates the access to the user's info (like his nickname). +During update, those user-specific data will be updated on all related media. + +## Optimizations + +Some indexes are useful for speeding up a MongoDB database and also in adding some controls. +In our case we have, for example, indexes on the user's email and his Twitter ID. + +All indexes are in the collection _system.indexes_ \ No newline at end of file diff --git a/Documentation/Design_pattern.md b/Documentation/Design_pattern.md new file mode 100644 index 0000000..a35ca1d --- /dev/null +++ b/Documentation/Design_pattern.md @@ -0,0 +1,26 @@ +Design Pattern +============== + +## Architecture + +* To build a powerful and scalable application, we chose to build a 3-tiers architecture. We have a Business part, +composed of the database and the API; and a Front part, which is a single angularJS application for the moment. + +* In the future, this architecture will allow us to easily build a mobile application or even a new website, +thanks to the API! + +## API + +* Our API has been made in Golang, a google language. We wanted a fast, new and low level language for the backend part, +and Golang is one of the best and mature language to do that. + +* We have Entities, who are the models. +* Routes-docs files are use for the routing part, plus they document the request. +* And there is controllers files, who consume entities to perform operations. + +We have a lot a additional things like filters, fixtures and unit tests. + +## Front + +* The Front is an angular application. We use MVVM pattern, so in our case the model is the Business part, +the View are the html files, and the ViewModel are the javascripts components files. \ No newline at end of file diff --git a/Documentation/Ergonomics.md b/Documentation/Ergonomics.md new file mode 100644 index 0000000..33d63b1 --- /dev/null +++ b/Documentation/Ergonomics.md @@ -0,0 +1,18 @@ +Ergonomics and ease of use +========================== + +## Design + +* We chose a dark theme design for clarity. It is a much better experience when watching a video to have a black background. +The primary color is a deep grey along with some white. Some information and titles are colored in a smooth dark blue. + +* The design is flat, simple and created to make the navigation intuitive and easy to use. + +## UX + +* The site has been designed to be clear. There is many picture and helpful words for a basic user. +With this, we think it's really easy to upload, watch and share video you like. + +* To prevent account accumulation, we have implement a Twitter account connexion. +Making the possibility to use your Twitter account to login to MewPipe. +Some others OpenID implementations are easy to implement, like Spotify, Github, Facebook, GPlus and so one... \ No newline at end of file diff --git a/Documentation/Install.md b/Documentation/Install.md new file mode 100644 index 0000000..32d2d19 --- /dev/null +++ b/Documentation/Install.md @@ -0,0 +1,83 @@ +Setup +===== + +## On a dev environment + +### Theory + +You must set an environment with Golang. In your GOPATH, create a directory supinfo/mewpipe and clone the project inside it. +Run ./fixtures.sh to get some example data. +Run ./test.sh to run the test suit on http://localhost:8080 +Run ./run.sh to run the server +Run ./build.sh to cross compile the project. + +### Practice + +On a fresh debian 8.1 install with sudo : + + #Install Golang + $ sudo apt-get install git + $ wget https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz + $ sudo tar -C /usr/local -xzf go1.4.2.linux-amd64.tar.gz + $ echo 'export PATH=$PATH:/usr/local/go/bin' | sudo tee -a /etc/profile + $ echo 'export GOPATH=/home/' | tee -a ~/.profile + $ source /etc/profile + $ source ~/.profile + $ rm go1.4.2.linux-amd64.tar.gz + $ mkdir -p src/supinfo + $ cd src/supinfo + $ git clone git@spider4all.com:4PJT-MewPipe mewpipe #Available on Github after the presentation @ Supinfo + $ mv 4PJT-MewPipe src/supinfo/mewpipe + $ cd mewpipe/ + $ go get + $ cp configs/local.ini.sample configs/local.ini + #Override vars from configs/base.ini in configs/local.ini like mongo connection string, database or http binding + $ ~/src/supinfo/mewpipe/run.sh + +## Deploy to prod + +### Build + +Run ./build.sh on dev environment, like : + + $ ./build.sh + Cross building, outputs are in /to/my/path/mewpipe/build + Building for linux on 386... + Compress build/mewpipe_linux_386.tar.bz2 + --------------------------- + Building for linux on amd64... + Compress build/mewpipe_linux_amd64.tar.bz2 + --------------------------- + Building for linux on arm... + Compress build/mewpipe_linux_arm.tar.bz2 + --------------------------- + Building for windows on 386... + Compress build/mewpipe_windows_386.tar.bz2 + --------------------------- + Building for windows on amd64... + Compress build/mewpipe_windows_amd64.tar.bz2 + --------------------------- + Building for darwin on 386... + Compress build/mewpipe_darwin_386.tar.bz2 + --------------------------- + Building for darwin on amd64... + Compress build/mewpipe_darwin_amd64.tar.bz2 + --------------------------- + +### Upload to Prod + +Choose the right build and upload it on your server or prod environment. Your right build is mewpipe_{Operating kernel}_{processor infrastructure}.tar.bz2 +In my case, on my mac OS X it's mewpipe_darwin_amd64.tar.bz2, on my debian it's mewpipe_linux_amd64.tar.bz2, on my Raspberry Pi it's mewpipe_linux_arm.tar.bz2 + +On my dev environment, I use the following command : + + $ scp build/mewpipe_linux_amd64.tar.bz2 debian@:mewpipe.tar.bz2 + + +On my prod server, I use + + $ tar -xjf mewpipe.tar.bz2 + $ rm mewpipe.tar.bz2 + $ chmod +x mewpipe + $ mewpipe --help # This show all variables you can overload on config.ini or pass in parameter. + $ ./mewpipe -config="config.ini" # You are free to deamonize this process. \ No newline at end of file diff --git a/Documentation/Suggested_algorithm.md b/Documentation/Suggested_algorithm.md new file mode 100644 index 0000000..b1ea4b0 --- /dev/null +++ b/Documentation/Suggested_algorithm.md @@ -0,0 +1,16 @@ +Suggested Video algorithm explanation +===================================== + +## Client needs + + On the home page of the website, the platform will suggest users some videos the most played and shared on Mewpipe. + +We have done that feature and its great. + +## Our purpose + +In the future, if the client wants an advanced suggestion mechanism, the dev'team can add a "Taxonomy" feature. +At upload, the user adds some tags related to the video, like "Cats" or "timelapse". +In a video, we may also create a list of videos that have the most related taxonomies, sorting them by views-count. +If we have less than _n_ videos with all of taxonomies, the list will be filled with videos +that matches _n-1_ taxonomies, _n-2_ taxonomies and so on... \ No newline at end of file diff --git a/README.md b/README.md index 91794bb..26c0beb 100644 --- a/README.md +++ b/README.md @@ -68,3 +68,10 @@ And with his [kind courtesy](https://twitter.com/aureliensalomon/status/60973022 * Run the app on the fly with ./run.sh * Run test suite with ./test.sh +## Documentation + +* [Setup](Documentation/Install.md) +* [Database structure and optimization](Documentation/Database.md) +* [Ergonomics and ease to use](Documentation/Ergonomics.md) +* [Suggested Video algorithm explanation](Documentation/Suggested_algorithm.md) +* [Design Pattern](Documentation/Design_pattern.md)