GstShark is an open-source project from RidgeRun that provides benchmarks and profiling tools for GStreamer 1.7.1 (and above). It includes tracers for generating debug information plus some tools to analyze the debug information. GstShark provides easy to use and useful tracers, paired with analysis tools to enable straightforward optimizations.
GstShark leverages GStreamer's built in tracing system and adds plotting tools to simplify the process of understanding the bottlenecks in your pipeline.
As part of the TAPPAS framework, we have expanded the available profiling mechanisms to include TAPPAS related tracers and a new plotting tool.
The profiling tool provides 3 general features that can be used to debug the pipeline:
- Tracers log printouts - At the most basic level, you should get printouts from the traces about the different measurements made. If you know what you are looking for, you may see it here at runtime. You can see the printouts in the debug file: GST_DEBUG_FILE=$TAPPAS_WORKSPACE/tappas_traces.log.
- Graphic visualization - Shown above, gst-shark can generate a pipeline graph that shows how elements are connected and what caps were negotiated between them. This is a very convenient feature to look at the pipeline in a more comfortable way. The graph is generated at runtime so it is a great way to see and debug how elements were actually connected and what formats the data ended up in.
- Plotting tool - A python script that generates a graph plot for each tracer metric enabled. This is a powerful tool to visualize each metric that can be used for deeper debugging.
Our docker image already contains GstShark! If you decide to not use our Docker image, our suggestion is to follow RidgeRun tutorial: GstShark
Note
These shortcuts are only available if you are using the docker tappas installation, Otherwise, please refer to the following section: Using GstShark in yocto-compiled images
As part of our creation of the Docker image, we copy some convinet shortcuts to GstShark:
vim ~/.bashrc# set gstreamer debug
gst_set_debug() {
export GST_SHARK_LOCATION=/tmp/profile
export GST_DEBUG="GST_TRACER:7"
export GST_DEBUG_FILE=$TAPPAS_WORKSPACE/tappas_traces.log
export GST_TRACERS="cpuusage;proctime;interlatency;scheduletime;bitrate;framerate;queuelevel;threadmonitor;numerator;buffer;detections;graphic"
export GST_DEBUG_NO_COLOR=1
echo 'Options for TRACERS:"'
echo 'export GST_TRACERS="cpuusage;proctime;interlatency;scheduletime;bitrate;framerate;queuelevel;threadmonitor;numerator;buffer;detections;graphic"'
}
# set trace to collect graphic data of gstreamer pipeline
gst_set_graphic() {
export GST_SHARK_LOCATION=/tmp/profile
export GST_DEBUG_DUMP_DOT_DIR=/tmp/
export GST_DEBUG="GST_TRACER:7"
export GST_TRACERS="graphic"
echo 'export GST_TRACERS="graphic"'
}
# unset gstreamer debug
gst_unset_debug() {
unset GST_TRACERS
unset GST_DEBUG
unset GST_DEBUG_DUMP_DOT_DIR
unset GST_SHARK_LOCATION
unset GST_DEBUG_FILE
unset GST_DEBUG_NO_COLOR
}
# plot the gst-shark dump files
gst_plot_debug() {
export GST_SHARK_LOCATION=/tmp/profile
split_traces_dir=$TAPPAS_WORKSPACE/tappas_traces_$(date +%d.%m.%Y_%H:%M:%S)
$TAPPAS_WORKSPACE/sources/gst-shark/scripts/graphics/split_traces.sh $split_traces_dir
python3 $TAPPAS_WORKSPACE/sources/gst-shark/scripts/graphics/plot_all_to_html.py -p $split_traces_dir
echo 'In order to plot the graphic pipeline graph, run:'
echo "dot $GST_SHARK_LOCATION/graphic/pipeline_<timestamp>.dot -T x11"
}Note that we added 4 functions: two sets, an unset, and a plot function. The set functions enable gst-shark by setting environment variables, the chief of which is GST_TRACERS. This enables the different trace hooks in the pipeline. The available tracers are listed in the echo command at the end of each set. You can enable any combination of the available tracers, just chain them together with a ';' (notice that the difference between gst_set_debug and gst_set_graphic is that gst_set_debug enables all tracers whereas gst_set_graphic only enables the graphic tracer that draws the pipeline graph). GST_SHARK_LOCATION and GST_DEBUG_DUMP_DOT_DIR set locations where the dump files are stored, the first sets where the tracer dumps are (used for gst-plot), and the latter where the dot file is saved (the graphic pipeline graph). Unset disables all tracers, and gst_plot_debug runs plot script.
Enable TAPPAS tracers:
Export the following environment variables:
export GST_SHARK_LOCATION=/tmp/profile export GST_DEBUG="GST_TRACER:7" export GST_DEBUG_NO_COLOR=1
Select the tracers by setting the GST_TRACERS environment variable to the list of tracers, seperated by ; as in the example:
export GST_TRACERS="scheduletime;bitrate;threadmonitor;numerator;buffer;detections"
You should export only some of the tracers, exporting too many tracers may fail the embedded device. All tracers explained: Understanding GstShark tappas plotted graphs
If you want to save the output to a file, run the following command before running the app:
export GST_DEBUG_FILE=<file_path>
If you want to plot the tracers output, use a strong machine (not an embedded device) with a full tappas installation and copy there the output file, then run the following commands:
export GST_DEBUG_FILE=<file_path> gst_plot_debug
Let’s say you have a gstreamer app you want to profile. Start by enabling gst-shark:
Then just run your app. You can see all kinds of tracer prints on the debug output file: GST_DEBUG_FILE=$TAPPAS_WORKSPACE/tappas_traces.log.
After you have run a gstreamer pipeline with tracers enabled, you can plot them using the plot script, just run:
gst_plot_debugIt will print to the console the path of the html file that contains the plots. You can open it in your browser. In addition it will print the command to open the pipeline graph. You can run it in a terminal to open the graph.
Each graph inspects a different metric of the pipeline, it is recommended to read more about what each one represents here:
- CPU Usage (cpuusage) - Measures the CPU usage every second. In multiprocessor systems this measurements are presented per core.
- Processing Time (proctime) - Measures the time an element takes to produce an output given the corresponding input.
- InterLatency (interlatency) - Measures the latency time at different points in the pipeline.
- Schedule Time (scheduling) - Measures the amount of time between two consecutive buffers in a sink pad.
- Buffer (buffer) - Prints information of every buffer that passes through every sink pad in the pipeline.
- Bitrate (bitrate) - Measures the current stream bitrate in bits per second.
- Framerate (framerate) - Measures the amount of frames that go through a src pad every second.
- Queue Level (queuelevel) - Measures the amount of data queued in every queue element in the pipeline.
- Thread Monitor (threadmonitor) - Measures the CPU usage of every thread in the pipeline.
- Numerator (numerator) - Numerates the buffers by setting the field "offset" of the buffer metadata. This trace is different from the others because it does not collect any data, it just numerates the buffers.
- Detections (detections) - Prints information about the objects detected in every buffer that passes through every pad in the pipeline. This trace only works with the TAPPAS framework since it collects the TAPPAS detection objects.
- Graphic (graphics) - Records a graphical representation of the current pipeline.
$ export GST_SHARK_FILE_BUFFERING=0With the no buffering mode every I/O operation is written as soon as possible.
The following command is an example of how to define the environment variable that will change the buffering mode to full buffering and the buffering size, this command uses a positive integer value for the size:
$ export GST_SHARK_FILE_BUFFERING=1024The individual element tracing, or filter parameter, allows the user to choose which elements get included in the tracing. The value to be set in the filter is a Glib Compatible Regular Expression, meaning that elements to be traced can be grouped by using a regex that matches with their name.
The filtering applies to the element name, NOT the factory. This is, if your element is specified as "identity name=myelem", you should refer to "myelem" and not to "identity"
Print the amount of frames that flow every 5 seconds through the different src pads in the pipe:
GST_TRACERS="framerate(period=5)" GST_DEBUG=GST_TRACER:7Print the amount of bits that flow every 3 seconds through the different src pads in the pipe:
GST_TRACERS="bitrate(period=3)" GST_DEBUG=GST_TRACER:7Print the amount of frames that flow every 5 seconds and bits that flow every 3 seconds through the different src pads in the pipe:
GST_TRACERS="framerate(period=5);bitrate(period=3)" GST_DEBUG=GST_TRACER:7Print the amount of frames that flow every 5 through the identity:
GST_TRACERS="framerate(period=5,filter=identity);bitrate(period=3)" GST_DEBUG=GST_TRACER:7Good luck, happy hunting.
gst-instruments is a set of performance profiling and data flow inspection tools for GStreamer pipelines.
gst-top-1.0at the start of the pipeline will analyze and profile the run. (gst-top-1.0 gst-launch-1.0 ! audiotestsrc ! autovideosink)gst-report-1.0- generates performance report for input trace file.gst-report-1.0 --dot gst-top.gsttracee | dot -Tsvg > perf.svg- generates performance graph in DOT format.