Merge branch 'kaltura:master' into master

.github/workflows/ci.yml

@@ -31,7 +31,7 @@ jobs:
             libssl-dev \
             libpcre3-dev \
             libxml2-dev \
-            wget \
+            curl \
             unzip \
             libopencore-amrwb0 \
             libopencore-amrnb0 \

README.md

@@ -3,6 +3,8 @@
 
 [Join the list of organizations using this video packager project](https://github.com/kaltura/nginx-vod-module/issues/730/).
 
+For live video streaming, please use [Media-Framework](https://github.com/kaltura/media-framework/).
+
 ### Features
 
 * On-the-fly repackaging of MP4 files to DASH, HDS, HLS, MSS
@@ -20,7 +22,7 @@
 
 * Fallback support for file not found in local/mapped modes (useful in multi-datacenter environments)
 
-* Video codecs: H264, H265 (DASH/HLS), VP8 (DASH), VP9 (DASH), AV1 (DASH)
+* Video codecs: H264, H265 (DASH/HLS), AV1 (DASH/HLS), VP8 (DASH), VP9 (DASH)
 
 * Audio codecs: AAC, MP3 (HLS/HDS/MSS), AC-3 (DASH/HLS), E-AC-3 (DASH/HLS), VORBIS (DASH), OPUS (DASH), FLAC (HLS), DTS (HLS)
 
@@ -124,7 +126,7 @@ C Macro Configurations:
 ### Installation
 
 #### RHEL/CentOS 6/7 RPM
-```
+```sh
 # rpm -ihv http://installrepo.kaltura.org/releases/kaltura-release.noarch.rpm
 # yum install kaltura-nginx
 ```
@@ -133,26 +135,26 @@ C Macro Configurations:
 *Ubuntu NOTE: before trying to install kaltura-nginx, you must also make sure the multiverse repo is enabled*
 
 For Debian Wheezy [7], Debian Jessie [8], Ubuntu 14.04 and 14.10, add this repo:
-```
+```sh
 # wget -O - http://installrepo.kaltura.org/repo/apt/debian/kaltura-deb-curr.gpg.key|apt-key add -
 # echo "deb [arch=amd64] http://installrepo.kaltura.org/repo/apt/debian propus main" > /etc/apt/sources.list.d/kaltura.list
 ```
 
 For Ubuntu 16.04, 16.10 add this repo:
-```
+```sh
 # wget -O - http://installrepo.kaltura.org/repo/apt/xenial/kaltura-deb-curr-256.gpg.key|apt-key add -
 # echo "deb [arch=amd64] http://installrepo.kaltura.org/repo/apt/xenial propus main" > /etc/apt/sources.list.d/kaltura.list
 ```
 
 For Ubuntu 20.04 add this repo:
-```
+```sh
 # wget -O - http://installrepo.kaltura.org/repo/aptn/focal/kaltura-deb-256.gpg.key|apt-key add -
 # echo "deb [arch=amd64] http://installrepo.kaltura.org/repo/aptn/focal quasar main" > /etc/apt/sources.list.d/kaltura.list
 ```
 
 
 Then install the kaltura-nginx package:
-```
+```sh
 # apt-get update
 # apt-get install kaltura-nginx
 ```
@@ -266,7 +268,7 @@ But first, a couple of definitions:
 #### Simple mapping
 
 The JSON below maps the request URI to a single MP4 file:
-```
+```json
 {
 	"sequences": [
 		{
@@ -287,7 +289,7 @@ possible to combine more complex JSONs using multi URL.
 #### Adaptive set
 
 As an alternative to using multi URL, an adaptive set can be defined via JSON:
-```
+```json
 {
 	"sequences": [
 		{
@@ -313,7 +315,7 @@ As an alternative to using multi URL, an adaptive set can be defined via JSON:
 #### Playlist
 
 The JSON below will play 35 seconds of video1 followed by 22 seconds of video2:
-```
+```json
 {
 	"durations": [ 35000, 22000 ],
 	"sequences": [
@@ -337,7 +339,7 @@ The JSON below will play 35 seconds of video1 followed by 22 seconds of video2:
 
 The JSON below takes video1, plays it at x1.5 and mixes the audio of the result with the audio of video2,
 after reducing it to 50% volume:
-```
+```json
 {
 	"sequences": [
 		{
@@ -375,7 +377,7 @@ after reducing it to 50% volume:
 The JSON below is a sample of a continuous live stream (=a live stream in which all videos have exactly the same encoding parameters).
 In practice, this JSON will have to be generated by some script, since it is time dependent.
 (see test/playlist.php for a sample implementation)
-```
+```json
 {
 	"playlistType": "live",
 	"discontinuity": false,
@@ -404,7 +406,7 @@ In practice, this JSON will have to be generated by some script, since it is tim
 The JSON below is a sample of a non-continuous live stream (=a live stream in which the videos have different encoding parameters).
 In practice, this JSON will have to be generated by some script, since it is time dependent 
 (see test/playlist.php for a sample implementation)
-```
+```json
 {
 	"playlistType": "live",
 	"discontinuity": true,
@@ -530,6 +532,8 @@ Optional fields:
 * `label` - a friendly string that identifies the sequence. If a language is specified,
 	a default label will be automatically derived by it - e.g. if language is `ita`, 
 	by default `italiano` will be used as the label.
+* `default` - a boolean that sets the value of the DEFAULT attribute of EXT-X-MEDIA tags using this sequence.
+	If not specified, the first EXT-X-MEDIA tag in each group returns DEFAULT=YES, while the others return DEFAULT=NO.
 * `bitrate` - an object that can be used to set the bitrate for the different media types,
 	in bits per second. For example, `{"v": 900000, "a": 64000}`. If the bitrate is not supplied,
 	nginx-vod-module will estimate it based on the last clip in the sequence.
@@ -569,6 +573,7 @@ Mandatory fields:
 	an empty captions file (useful in case only some videos in a playlist have captions)
 
 Optional fields:
+* `id` - a string that identifies the source clip
 * `sourceType` - sets the interface that should be used to read the MP4 file, allowed values are:
 	`file` and `http`. By default, the module uses `http` if `vod_remote_upstream_location` is set,
 	and `file` otherwise.
@@ -678,7 +683,7 @@ of the origin will be able to bypass the CDN token enforcement. If using Akamai,
 be accomplished using https://github.com/refractalize/nginx_mod_akamai_g2o.
 For other CDNs, it may be possible to configure the CDN to send a secret header to the origin
 and then simply enforce the header using an nginx if statement:
-```
+```c
 		if ($http_x_secret_origin_header != "secret value") {
 			return 403;
 		}
@@ -740,7 +745,7 @@ The `vod_drm_upstream_location` parameter specifies an nginx location that is us
 and the request uri is configured using `vod_drm_request_uri` (this parameter can include nginx variables). 
 The response of the DRM server is a JSON, with the following format:
 
-```
+```json
 [{
 	"pssh": [{
 			"data": "CAESEGMyZjg2MTczN2NjNGYzODIaB2thbHR1cmEiCjBfbmptaWlwbXAqBVNEX0hE", 
@@ -761,7 +766,7 @@ The response of the DRM server is a JSON, with the following format:
 ##### Sample configurations
 
 Apple FairPlay HLS:
-```
+```nginx
 location ~ ^/fpshls/p/\d+/(sp/\d+/)?serveFlavor/entryId/([^/]+)/(.*) {
 	vod hls;
 	vod_hls_encryption_method sample-aes;
@@ -782,7 +787,7 @@ location ~ ^/fpshls/p/\d+/(sp/\d+/)?serveFlavor/entryId/([^/]+)/(.*) {
 ```
 
 Common Encryption HLS:
-```
+```nginx
 location ~ ^/cenchls/p/\d+/(sp/\d+/)?serveFlavor/entryId/([^/]+)/(.*) {
 	vod hls;
 	vod_hls_encryption_method sample-aes-cenc;
@@ -1687,8 +1692,19 @@ padding is added as needed.
 * **default**: `off`
 * **context**: `http`, `server`, `location`
 
-When enabled, an ID3 TEXT frame will be outputted in each TS segment, containing a JSON with the absolute segment timestamp.
-The timestamp is measured in milliseconds since the epoch (unixtime x 1000), the JSON structure is: `{"timestamp":1459779115000}`
+When enabled, an ID3 TEXT frame is outputted in each TS segment.
+The content of the ID3 TEXT frame can be set using the directive `vod_hls_mpegts_id3_data`.
+
+#### vod_hls_mpegts_id3_data
+* **syntax**: `vod_hls_mpegts_id3_data string`
+* **default**: `{"timestamp":$vod_segment_time,"sequenceId":"$vod_sequence_id"}`
+* **context**: `http`, `server`, `location`
+
+Sets the data of the ID3 TEXT frame outputted in each TS segment, when `vod_hls_mpegts_output_id3_timestamps` is set to `on`.
+When the directive is not set, the ID3 frames contain by default a JSON object of the format `{"timestamp":1459779115000,"sequenceId":"{id}"}`:
+- `timestamp` - an absolute time measured in milliseconds since the epoch (unixtime x 1000).
+- `sequenceId` - the id field of the sequence object, as specified in the mapping JSON. The field is omitted when the sequence id is empty / not specified in the mapping JSON.
+The parameter value can contain variables.
 
 #### vod_hls_mpegts_align_pts
 * **syntax**: `vod_hls_mpegts_align_pts on/off`
@@ -1698,6 +1714,13 @@ The timestamp is measured in milliseconds since the epoch (unixtime x 1000), the
 When enabled, the module will shift back the dts timestamps by the pts delay of the initial frame.
 This can help keep the pts timestamps aligned across multiple renditions.
 
+#### vod_hls_encryption_output_iv
+* **syntax**: `vod_hls_encryption_output_iv on/off`
+* **default**: `off`
+* **context**: `http`, `server`, `location`
+
+When enabled, the module outputs the `IV` attribute in returned `#EXT-X-KEY` tags.
+
 ### Configuration directives - MSS
 
 #### vod_mss_manifest_file_name_prefix
@@ -1819,6 +1842,7 @@ The module adds the following nginx variables:
 	`EXPIRED` - the current server time is larger than `expirationTime`
 	`ALLOC_FAILED` - the module failed to allocate memory
 	`UNEXPECTED` - a scenario that is not supposed to happen, most likely a bug in the module
+* `$vod_segment_time` - for segment requests, contains the absolute timestamp of the first frame in the segment, measured in milliseconds since the epoch (unixtime x 1000).
 * `$vod_segment_duration` - for segment requests, contains the duration of the segment in milliseconds
 * `$vod_frames_bytes_read` - for segment requests, total number of bytes read while processing media frames
 
@@ -1827,7 +1851,7 @@ Note: Configuration directives that can accept variables are explicitly marked a
 ### Sample configurations
 
 #### Local configuration
-
+```nginx
 	http {
 		upstream fallback {
 			server fallback.kaltura.com:80;
@@ -1873,9 +1897,10 @@ Note: Configuration directives that can accept variables are explicitly marked a
 			}
 		}
 	}
+```
 
 #### Mapped configuration
-
+```nginx
 	http {
 		upstream kalapi {
 			server www.kaltura.com:80;
@@ -1936,9 +1961,10 @@ Note: Configuration directives that can accept variables are explicitly marked a
 			}
 		}
 	}
+```
 
 #### Mapped + Remote configuration
-
+```nginx
 	http {
 		upstream jsonupstream {
 			server jsonserver:80;
@@ -1992,9 +2018,10 @@ Note: Configuration directives that can accept variables are explicitly marked a
 			}
 		}
 	}
+```
 
 Set it up so that http://jsonserver:80/test.json returns the following JSON:
-
+```json
 	{
 		"sequences": [{
 			"clips": [{
@@ -2003,11 +2030,12 @@ Set it up so that http://jsonserver:80/test.json returns the following JSON:
 			}]
 		}]
 	}
+```
 
 And use this stream URL - http://nginx-vod-server/hls/test.json/master.m3u8
 
 #### Remote configuration
-
+```nginx
 	http {
 		upstream kalapi {
 			server www.kaltura.com:80;
@@ -2045,7 +2073,7 @@ And use this stream URL - http://nginx-vod-server/hls/test.json/master.m3u8
 			}
 		}
 	}
-
+```
 ### Copyright & License
 
 All code in this project is released under the [AGPLv3 license](http://www.gnu.org/licenses/agpl-3.0.html) unless a different license for a particular library is specified in the applicable library path. 

ci_build.sh

@@ -1,18 +1,21 @@
 #!/bin/sh
-set -o nounset                              # Treat unset variables as an error
+set -eo nounset                              # Treat unset variables as an error
 
-NGINX_VERSION=1.23.0
-NGINX_URI="http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz"
+BASE_DOWNLOAD_URI=http://nginx.org/download
+NGINX_VERSION=`curl -L "$BASE_DOWNLOAD_URI" |
+   grep -oP 'href="nginx-\K[0-9]+\.[0-9]+\.[0-9]+' |
+   sort -t. -rn -k1,1 -k2,2 -k3,3 | head -1`
+NGINX_URI="$BASE_DOWNLOAD_URI/nginx-$NGINX_VERSION.tar.gz"
 
-
-if [ ! -x "`which wget 2>/dev/null`" ];then
-        echo "Need to install wget."
+if [ ! -x "`which curl 2>/dev/null`" ];then
+        echo "Need to install curl."
         exit 2
 fi
+
 mkdir -p /tmp/builddir/nginx-$NGINX_VERSION
 cp -r . /tmp/builddir/nginx-$NGINX_VERSION/nginx-vod-module
 cd /tmp/builddir
-wget $NGINX_URI -O kaltura-nginx-$NGINX_VERSION.tar.gz
+curl $NGINX_URI > kaltura-nginx-$NGINX_VERSION.tar.gz
 tar zxf kaltura-nginx-$NGINX_VERSION.tar.gz
 mv nginx-$NGINX_VERSION nginx
 cd nginx
@@ -55,6 +58,6 @@ export LD_LIBRARY_PATH LIBRARY_PATH C_INCLUDE_PATH
         --with-ipv6 \
         --with-debug \
         --with-threads \
-        --with-cc-opt="-O3 -mpopcnt" \
+	--with-cc-opt="-O3 -mpopcnt" \
         $*
 make -j $(nproc)

config

@@ -213,7 +213,7 @@ ngx_feature_incs="#include <libavfilter/avfilter.h>
                   #include <libavfilter/buffersink.h>"
 ngx_feature_path=
 ngx_feature_libs="$LIB_AV_FILTER"
-ngx_feature_test="avfilter_register_all()"
+ngx_feature_test="avfilter_get_by_name(NULL)"
 . auto/feature
 
 if [ $ngx_found = yes ]; then
@@ -399,6 +399,7 @@ VOD_SRCS="$VOD_SRCS                                           \
           $ngx_addon_dir/vod/language_code.c                  \
           $ngx_addon_dir/vod/manifest_utils.c                 \
           $ngx_addon_dir/vod/media_format.c                   \
+          $ngx_addon_dir/vod/media_set.c                      \
           $ngx_addon_dir/vod/media_set_parser.c               \
           $ngx_addon_dir/vod/mkv/ebml.c                       \
           $ngx_addon_dir/vod/mkv/mkv_builder.c                \