Tag Archives: encoding

Fix Linux filename encodings with Python

Posted on by
6

We recently deployed a web based system to old Red Hat installation still using latin-1 as primary encoding for its filesystem. This caused headscratching as we were using Ubuntu during the development which uses UTF-8 encoding for filenames. Files were copied to the production server using rsync. When ls command was used in a terminal everything seemed be fine, because terminal itself was configured to UTF-8 and thus decoded the filenames when outputted to the terminal. However the web server was unable to serve images containing Finnish ä and ö characters in filenames.

After the headscratching was over we had figured out that renaming filenames to use latin-1 encoding makes the web server serve them fine. If it were a fresh start I’d rather configure the server itself to use now prevailent UTF-8, but because the infrastructure was shared with other, older, projects this was out of the question.

Thus, the script below was created to address the filename encoding problem (syntax highlighting available on the orignal publication):

#!/usr/bin/python
"""

    Recursively fix filename encoding problems

    http://www.opensourcehacker.com, MIT licensed

"""

import os

# Source filename encoding
FROM="utf-8"

# Target filename encoding
TO="latin-1"

# Current working directory
PATH=os.getcwd()

for root, dirs, files in os.walk(PATH):

    # Assume files are 8-bit strings in the native encoding
    # of the system which we cannot know
    for f in files:
        try:
            decoded = f.decode(FROM)
        except UnicodeDecodeError:
            print "Cannot decode:" + f
            continue

        fullpath = os.path.join(root, f)

        newpath = os.path.join(root, decoded.encode(TO))

        if newpath != fullpath:
            print "Renaming:" + fullpath + " to:" + newpath
            os.rename(fullpath, newpath)

I still don’t know how to verify / check / guess which encoding Linux file systems are using (if not UTF-8…), so someone please enlight me. In this case we learnt this by a smart guess and testing: we created two filenames, one name encoded with UTF-8 and one encoded with Latin-1 and see which file the web server was giving for us. I am also unsure whether the fact that files were orignally kept in Subversion repository and committed there from Windows had anything to do with the problem.

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

How to encode h264 video files for Nokia Series 60 standalone playback

Posted on by
3

Bored with Spiderman 3 which came with your Nokia N95 8 GB? This guide shortly tells how to get movies into your N95 on Ubuntu Linux using ffmpeg video encoder. The aim is to encode video suitable for playback from Nokia N-series (N95, N78, others) mobile phone memory card. We use h264 + AAC codecs which provides the best quality/compression rate for Nokia phones currently.

Ubuntu does not distribute proprietary codes. First thing you need to do is to rebuild ffmpeg.  Since Ubuntu 8.04 Hardy Heron ships with ffmpeg from 2007, which is aeons old in video codec years, you need to build libx264 and ffmpeg from SVN sources. Here are detailed, valid, instructions. Note that FFMPEG trunk is not currently stable (September 2008), so you need to use revision 15261 which needs this little patch. Indeed, this is a very difficult month to start your career in the dark world of video encoders.

To make it legal and support open source codec development,  please pay for your codecs.

Then we use this guide by Robert Swain. We have a tiny sub 2,4″ screen, we do not care about the quality and do one pass encoding. By empirical research, I have found that the following MPEG-4 profile parameters are compatible with N95 8 GB and provide the optimal result. You can vary video and audio bitrate depending on your taste.

Here is a script which recursivelu encodes all detected video files suitable for mobile format:

#!/bin/sh
#
# Optimal movie encoding for Nokia N-series mobile phones
#
# Copyright 2008 Red Innovation Ltd. 
#
# Say hi if you find this useful.
# We do some professional mobile video publishing, so if you
# need a helping hand please call us.
#
# Usage: Run encode.sh in any folder and all video files are recursively converted to mobile phone suitable format
#
# Note: We expect all the source material be in 16:9 aspect ration
#
# Also see http://www.nseries.com/index.html#l=support,search,faq,general,video%20encoding,53848
#

VIDEO_BITRATE=300k

AUDIO_BITRATE=72k

# Assume locally build ffmpeg + x264 in /usr/local/bin
# http://ubuntuforums.org/showthread.php?t=786095
export LD_LIBRARY_PATH=/usr/local/lib

# Search all source AVI, MPG and WMV video files
# Place all encoded files to the same folder with the source, with added .mp4 extension
find . -iname "*.avi" -or -iname "*.wmv" -or -iname "*.mpg" | while read src ; do
        srcfile=`basename "$src"`
	srcfolder=`dirname "$src"`
	dstfile="$srcfolder"/"$srcfile".mp4

	# The magical string!
	# Size and cropping is for 16:9 source material, so that 320:240 display will have black bars.
	# Fex pixels off... note that h264 sizes must be multiplies of 16, use 256x144 for streaming
	# N95 RealMedia player does not seem to respect MPEG-4 embedded aspect ration info.
	/usr/local/bin/ffmpeg -y -i "$srcfile" -acodec libfaac -ab $AUDIO_BITRATE -s 320x176 -aspect 16:9 -vcodec libx264 -b $VIDEO_BITRATE -qcomp 0.6 -qmin 16 -qmax 51 -qdiff 4 -flags +loop -cmp +chroma -subq 7 -refs 6 -g 250 -keyint_min 25 -rc_eq 'blurCplx^(1-qComp)' -sc_threshold 40 -me_range 12 -i_qfactor 0.71 -directpred 3 "$dstfile"

done