The goal of this walkthrough is to create a pulsating fade effect that can be continuously applied to an input stream where the duration is unknown, like a live source.

geq to the rescue

The geq filter allows the usage of an equation to be applied to each pixel. We can utilized this filter with the T variable which references the time of the current frame.

For our example, we will combine it with a simple sin to get a cyclic effect with a multiplier.

geq=a='abs(255*sin(T))'

• a / alpha_exprcannot be set alone. Either lum or one of the r, g, b must be set along side. A hacky solution is to set rgb to its original value via r=r(X,Y)
• abs is used here to simplify the sin‘s negative cycle
• The layer must be converted to have a alpha channel so that the alpha channel can be changed

Results

Combining some knowledge from the previous post on Blur Out Videos with FFmpeg, we get the end result. This takes in 1 input source, generates the blurred version of it, then adjusts the alpha according to time, then overlays the results on the base.

不久之前，RD的夥伴們完成了一場精彩絕倫的分享！ 這次有四位同仁分享覺得有趣的主題，分別為：

• Lego CI/CD with github action
• The wet codebase
• Enlarge eyes by Apple face detection and Metal shader
• Smooth skin principle and implementation

快來看看這次的Presentations吧！

Enlarge eyes by Apple face detection and Metal shader

• Introduce Metal and compare to OpenGL ES
• Introduce Vision
• Enlarge eyes algorithm

Smooth skin principle and implementation

本篇介紹 GPUImage 中一個常用的美肌濾鏡，並深入解釋其中使用高斯模糊濾鏡的原因及原理。

• Overview of filter pipeline
• Box blur vs. Gaussian blur and how to implement with shader
• Why Gaussian blur is suitable for smooth skin (fourier transform, low-pass filter)
• Filter pipeline principle summary
• Direction to improve performance and combine with face detection

Lego CI/CD with Github Actions

Using features on Github (Github Action, Github App, Github Deployment) implement a GitOps flow which make application developer and DevOps engineer cowork with each other easier, and how this flow improve the development experience.

Presentations Day #1 was originally published in SWAG on Medium, where people are continuing the conversation by highlighting and responding to this story.

The problem

Unfortunately, scale_npp produced corrupt video when used.

This turns out that the drivers install with the daemonset provided by GKE is version 410.79 and has some problems with NVIDIA T4 GPUs. Running the same commands on a NVIDIA Quadro RTX 5000 with the same drivers produced non-corrupt video.

The incompatibility seems to be when scale_npp is actually needs to do scaling. If the source dimensions equal the output dimensions, the video is non-corrupt. When the source dimensions are not equal the output dimensions, the video is corrupt.

The Fix

Taking a look at the daemonset, the driver installation process is

1. Attempt to download and install a precompiled driver
2. Fallback to compiling and installing (This always errored)

The precompiled driver was hardcoded to a preformated download location on a Google Cloud Storage bucket. It would take the region and the default driver version specified in a script and attempted to download it.

Fortunately, the driver version could be configured from the env of the daemonset, but exact driver version would needed to be provided.

Running gsutil ls gs://nvidia-drivers-asia-public/tesla , you will be able to list the downloadable drivers by version number.

gs://nvidia-drivers-asia-public/tesla/384.183/
gs://nvidia-drivers-asia-public/tesla/390.116/
gs://nvidia-drivers-asia-public/tesla/396.26/
gs://nvidia-drivers-asia-public/tesla/396.37/
gs://nvidia-drivers-asia-public/tesla/396.44/
gs://nvidia-drivers-asia-public/tesla/396.82/
gs://nvidia-drivers-asia-public/tesla/410.104/
gs://nvidia-drivers-asia-public/tesla/410.72/
gs://nvidia-drivers-asia-public/tesla/410.79/
gs://nvidia-drivers-asia-public/tesla/418.126.02/
gs://nvidia-drivers-asia-public/tesla/418.40.04/
gs://nvidia-drivers-asia-public/tesla/418.67/
gs://nvidia-drivers-asia-public/tesla/418.87.00/
gs://nvidia-drivers-asia-public/tesla/418.87.01/
gs://nvidia-drivers-asia-public/tesla/440.64.00/

After choosing the driver version you want, 440.64.00 in our case, make a copy of the daemonset and set the the environment variableNVIDIA_DRIVER_VERSION=440.64.00 and re-install the daemonset.

Harnessing filter_complex

FFmpeg’s filter_complex works in a similar fashion as Unix pipes. Take a input, modify, output, then rinse and repeat. A filtergraph contains one or more filterchains, each which contains a certain order of filters to be applied to the input source. Like Unix pipes, you can get pretty creative and generate some great results.

Splitting Outputs

First, we will split a single input into 2 outputs with scaled resolutions.

• 0:v: Select the first input source’s videostream only
• split=2: Split the stream into 2 and store them as 360pand 720p
• [360p]scale=-2:360[360p] and [720p]scale=-2:720[720p] scales the input to 360px and 720pxheight respectively while preserving aspect ratio and sends the output back to the same input name. The -2 is to tell the scaler to scale to a even number as some formats do not support a odd number of pixels
• -map "[360p]" 360p.mp4 and -map "[720p]" 720p.mp4 take each respective output and encodes it to a file.

The result will be 2 files with different resolutions. Keep in mind, all outputs created in the filtergraph must be connected to an output.

Blurring

The next component is to generate a blurred video. There are many blurring algorithms available, but for this task, we will be using boxblur. This blur has 3 tunable settings each with radius (box radius to apply to the frame) and power (how many times to apply to the frame):

• luma (luma_radius and luma_power): Brightness
• chroma (chroma_radius and chroma_power): Color
• alpha (alpha_radius and alpha_power): Transparency

Fading In

The last part we need is a fade. This is filter is fairly simple as it takes a start and a end with either a fade in or fade out. It supports both frames and time/duration. For our case, we will use the time/duration and skip calculating frames.

Peanut Butter Jelly Time

Now that we have all the tools we need, let’s put this all together. The plan is to put each of the filters above to create a Blur Out effect. A Blur Out is essentially 2 videos (the original and a blurred version) overlayed on top of each other with the one of the videos fading.

For this example, we will choose the original video as the baseand the blurred video as the fade in.

1. We will will need to split the input into 2, base and blurred, so we use the split filter
2. Next we need to generate a blurred version using boxblur
3. Taking the blurred input, we add a fade in start time 1s and duration 3s. The extra alpha=1 mentioned here is to allow the video to be transparent when faded out else the background color will be black
4. The final step is to use overlay to put input sources on top of each other (like an onion). The alpha=1 from above would then allow the base to show through the blurred layer as it is being faded in.

References

• FFmpeg Filtering Guide

Blur Out Videos with FFmpeg was originally published in SWAG on Medium, where people are continuing the conversation by highlighting and responding to this story.

Compiling

To get python-rapidjson to compile with SIMD, we need to define one of the SIMD macros, either RAPIDJSON_SSE2, RAPIDJSON_SSE42, or RAPIDJSON_NEON.

The chosen flag would then need to be passed to pip during install via CFLAGS. Depending on the flag, you would have to pass some addition options.

• SSE2: CFLAGS="-DRAPIDJSON_SSE2=1"
• SSE4.2: CFLAGS="-DRAPIDJSON_SSE42=1 -msse4.2"

One-liner to re-install the currently installed version as the SIMD version.

CFLAGS="-DRAPIDJSON_SSE42=1 -msse4.2" pip -v install -force-reinstall -no-binary python-rapidjson $(pip freeze | grep python-rapidjson)

So what’s available?

There are many solutions available and is dependent on use case, but there are mainly 2 options, RTMP and DASH/HLS. While HLS is an option and having the much larger market share compared to DASH, DASH is vastly superior in terms of feature offering with DASH being able to support the majority of features of HLS. Many workflows are also switching to or dual supporting DASH with HLS.

What’s the difference?

RMTP

• Persistent TCP connection
  Requiring a persistent TCP connections is a double edged sword. It removes the connection overhead compared to a segment based delivery protocol over HTTP.
• Low Latency
  Being a pushed based delivery allows RTMP to achieve very low latency with potential to go as low as 0.5s.

DASH

• HTTP Based
  Being a segment based delivery over HTTP, this is huge plus(mostly). This means that DASH can take advantage of existing HTTP knowledge and infrastructure and any advancement in the HTTP spec. It can also take advantage of persistent connections in HTTP/1.1 or HTTP/2.0.
• Adaptive Streaming
  One of DASH’s main goals was to support multiple streams and the ability to seamlessly switch between them. The purpose for this was to support multiple bit rates so that the client can use the best fit for their case, reducing bandwidth requirements.
• Wider Container/Codec Support
  DASH is designed to be flexible, supporting MP4 and MPEG-TS and newer codecs such as HEVC and AV1.
• Discontinuity
  While mainly used for ad insertion, it is very straight forward to break a stream in progress.
• Content Protection
  Applications wanting to control content on devices have Fairplay, Widevine, PlayReady
• Controlling Group
  Unlike RTMP where Adobe is the sole developer, DASH was created from the cooperation between many groups (Google, Apple, Microsoft to name a few) and not owned by any single company.

Roadblocks

RTMP, being a proprietary protocol developed by Adobe(Macromedia) which uses FLV, requires Flash to be usable on web browsers. With Flash being long gone, there is no straightforward way to support RMTP in browsers. Instead, support in modern browsers is via FLV over HTTP (via flv.js) resulting in increased management overhead. Technology momentum is also very important. Like with any technology, it should improve over time, but advancements to RMTP are nearly non-existent and has lost momentum over the years to other protocols like DASH, HLS and WebRTC.

DASH also has areas that need to be addressed with latency being one of the biggest. For applications that require near realtime/ultra low latency, DASH is not quite there yet. CMAF aims to improve this by introducing a specification on the encoding and transfer process.

Paving the Road Ahead

HTTP/2.0 Server Push
While Server Push being relatively unused, in terms of video delivery, it can help move DASH even closer to RMTP in terms of latency by pushing segments immediately after encoding has finished.

QUIC (aka HTTP/3.0)
As mentioned above, being HTTP is a huge plus. With QUIC being standardized as HTTP/3.0, this mean DASH will in-essence receive free UDP support in addition to better network switching and a more efficient TLS connection.

To be fair, there have been some work done by QiNiu (a China based RTMP CDN) to integrate RTMP over QUIC.

CMAF Low Latency
Common Media Application Format for Low Latency is a encoding and transfer specification aimed to reduce latency by essentially sending IDRs as soon as they are encoded without waiting for the entire segment to encode.

Thoughts

While RMTP still have clear use cases in publishing and low latency viewing, it is quickly being replaced. Without a clear roadmap for RTMP and with DASH closing the gap on low latency, it seems very likely that DASH will come out the solution of choice in the coming years.

References

• https://www.encoding.com/files/2018-Global-Media-Formats-Report.pdf
• https://blogs.akamai.com/2018/10/best-practices-for-ultra-low-latency-streaming-using-chunked-encoded-and-chunk-transferred-cmaf.html
• https://bitmovin.com/cmaf-low-latency-streaming

Moving forward with Live Stream Video Delivery was originally published in SWAG on Medium, where people are continuing the conversation by highlighting and responding to this story.

The solution I took was to add a XFS support node label at node pool creation time using OS image ubuntu. While this sort of works, there are a couple problems.

• The need to remember to add a XFS node label when creating a new node pool
• The assumption that ubuntuhas xfsprogs installed
• The assumption that cos does not have xfsprogs installed. (It doesn’t right now but you never know)

Detecting XFS host support

Since provisioning a PersistentVolume is done on the host before the Pod is running, this complicates things as the support would need to be for the host, but the detection done in a Pod.

For this, we will use nsenter. nsenter allows running a process in a different namespace, in our case, the host. For us to properly use nsenter, we will need to set hostPID:true and priviledged: true, allowing us to break out of the pod into the host.

hostPID: true
volumes:
- name: tmp
  emptyDir: {}
initContainers:
- name: detect
  image: wardsco/nsenter
  command: ["sh", "-eo", "pipefail", "-c"]
  args: ["nsenter -t 1 -m -u -i -n -p -- sh -c 'command -v xfs_mkfile' && touch /tmp/xfs_mkfile || true"]
  securityContext:
    privileged: true
  volumeMounts:
  - name: tmp
    mountPath: /tmp/

The command command -v xfs_mkfile gets run on the host which detects if the command xfs_mkfile is available.

Labeling the node for support

The usage of the /tmp volume mount is to pass the results of detection into the labeling container. This is to drop the priviledged: true as soon as possible.

initContainers
- name: label
  image: wardsco/kubectl:1.11
  command: ["sh", "-eo", "pipefail", "-c"]
  args: ["kubectl label node --overwrite $NODE_NAME fs.type/xfs=$(test -e /tmp/xfs_mkfile && echo 'true' || echo 'false'"]
  env:
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  volumeMounts:
  - name: tmp
    mountPath: /tmp/
    readOnly: true

By using the Downward API, we can pass in the node name to the pod. This container will then label the node it is on with the label fs.type/xfs=true/false indicating support. With this, you can schedule pods with nodeAffinity.

Note: kubectl label node requires extra permissions which won’t be covered here. As a shortcut, setting spec.serviceAccountName: node-controller in the kube-system namespace provides these permissions.

Putting it together

The solution at the time was to use a Daemonset running a startup script with gcr.io/google-containers/startup-script:v1.

There are a couple of areas that could be improved

• hostPID and securityContext seemed excessive
• No checks if the setting actually changed
• gcr.io/google-containers/startup-script:v1 is a relatively large image
• Timing conflicts with pod scheduling

hostPID and securityContext

Instead of using hostPID and priviledged: true, we can mount the host’s /sysinto the pod as a volume.

volumes:
- name: sys
  hostPath:
    path: /sys

volumeMounts:
- name: sys
  mountPath: /rootfs/sys

Checking if settings applied

This part is straight forward. We simply grep the property and return an appropriate exit code.

grep -q -F [never] /sys/kernel/mm/transparent_hugepage/enabled
grep -q -F [never] /sys/kernel/mm/transparent_hugepage/defrag

Large Images

This one is not a critical problem. gcr.io/google-containers/startup-scriptis 12.5MB, but since we are essentially just running a shell script, it can be changed to a slimmer image, like busyboxwhich has an image size of 1.15MB. Of course busybox is lacking the startup functionality of gcr.io/google-containers/startup-script. For this we can utilize initContainers which were unavailable at the time.

Pod Scheduling Conflicts

This problem is referring to a dependency conflict where redis or mongo can be scheduled on a node where the kernel-tuner has not yet completed. Since the process was started before the setting was applied, it will not receive the updated kernel settings and would need a restart.

For this problem, we can use labels on nodes in conjunction with nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution. This solution will pend the pod scheduling until a node with the proper labels exist.

To label a node within a pod, there are some prerequisites.

• kubectl label node needs RBAC permission (skip if it is not required). For my case, I used the service account node-controller that is created by default on kube-systemnamespace on GKE by setting serviceAccountName: node-controller
• Pod needs to know the node name it lives on via Downward API

initContainers:
- name: label-node
  image: swaglive/kubectl:1.11
  command: ["kubectl"]
  args: ["label", "node", "--overwrite", "$(NODE_NAME)", "sysctl/mm.transparent_hugepage.enabled=never", "sysctl/mm.transparent_hugepage.defrag=never"]
  env:
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

Now to add the label restriction to pods that need it.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: sysctl/mm.transparent_hugepage.enabled
          operator: In
          values:
          - "never"
        - key: sysctl/mm.transparent_hugepage.defrag
          operator: In
          values:
          - "never"

Putting it all together

Using it with a redisdeployment

• Cluster Role Bindings

tl;dr Use empty volume, initContainers, and subPath to copy and mount kubectl.

The Why

I needed access to the kubernetes API from within a pod so that the pod can self label itself.

For example, I am currently working with redis and redis-sentinel. When sentinel triggers a reconfigure script, I want the pod to re-label itself to role=masteror role=slave. I didn’t want to create a custom redis image that includes kubectl as it would be another component to maintain.

Also, what if I needed to work with other images requiring kubectl? Seemed like alot of maintenance going the custom image route.

The How

First, create an empty volume to hold the kubectl binary.

volumes:
- name: kubectl
  emptyDir: {}

Next, using initContainers, copy out the kubectlbinary from a docker image into the volume. In this case, allanlei/kubectlis an image containing a static binary from kubernetes.

initContainers:
- name: install-kubectl
  image: allanlei/kubectl
  volumeMounts:
  - name: kubectl
    mountPath: /data
  command: ["cp", "/usr/local/bin/kubectl", "/data/kubectl"]

Finally, mount the kubectl volume into the container using subPath. If you don’t use subPath, then the entire mount path will get overriden or gets mounted as a directory, which is not the goal. subPath allows us to specify certain paths in the volume to be mounted.

volumeMounts:
- name: kubectl
  subPath: kubectl
  mountPath: /usr/local/bin/kubectl

You’re ready to go! The container is now able to run kubectl which is automatically setup via Service Accounts.

Full Example:

Single Line

For very simple stuff, a single command using the default alpine image.

Mounted Volume

For more complex crontabs, mount a crontab file!

Compiled Docker Image

Compiling your own image with an entrypoint makes it cleaner to run.