Martin Fitzpatrick - tutorials

Getting Started With Git and GitHub in Your Python Projects — Version-Controlling Your Python Projects With Git and GitHub

2023-03-20T06:00:00+00:00

Using a version control system (VCS) is crucial for any software development project. These systems allow developers to track changes to the project's codebase over time, removing the need to keep multiple copies of the project folder.

VCSs also facilitate experimenting with new features and ideas without breaking existing functionality in a given project. They also enable collaboration with other developers that can contribute code, documentation, and more.

In this article, we'll learn about Git, the most popular VCS out there. We'll learn everything we need to get started with this VCS and start creating our own repositories. We'll also learn how to publish those repositories to GitHub, another popular tool among developers nowadays.

Installing and Setting Up Git

To use Git in our coding projects, we first need to install it on our computer. To do this, we need to navigate to Git's download page and choose the appropriate installer for our operating system. Once we've downloaded the installer, we need to run it and follow the on-screen instructions.

We can check if everything is working correctly by opening a terminal or command-line window and running git --version.

Once we've confirmed the successful installation, we should provide Git with some personal information. You'll only need to do this once for every computer. Now go ahead and run the following commands with your own information:

shell

$ git config --global user.name <"YOUR NAME">
$ git config --global user.email <name@email.com>

The first command adds your full name to Git's config file. The second command adds your email. Git will use this information in all your repositories.

If you publish your projects to a remote server like GitHub, then your email address will be visible to anyone with access to that repository. If you don't want to expose your email address this way, then you should create a separate email address to use with Git.

As you'll learn in a moment, Git uses the concept of branches to manage its repositories. A branch is a copy of your project's folder at a given time in the development cycle. The default branch of new repositories is named either master or main, depending on your current version of Git.

You can change the name of the default branch by running the following command:

shell

$ git config --global init.defaultBranch <branch_name>

This command will set the name of Git's default branch to branch_name. Remember that this is just a placeholder name. You need to provide a suitable name for your installation.

Another useful setting is the default text editor Git will use to type in commit messages and other messages in your repo. For example, if you use an editor like Visual Studio Code, then you can configure Git to use it:

shell

# Visual Studio Code
$ git config --global core.editor "code --wait"

With this command, we tell Git to use VS Code to process commit messages and any other message we need to enter through Git.

Finally, to inspect the changes we've made to Git's configuration files, we can run the following command:

shell

$ git config --global -e

This command will open the global .gitconfig file in our default editor. There, we can fix any error we have made or add new settings. Then we just need to save the file and close it.

Understanding How Git Works

Git works by allowing us to take a snapshot of the current state of all the files in our project's folder. Each time we save one of those snapshots, we make a Git commit. Then the cycle starts again, and Git creates new snapshots, showing how our project looked like at any moment.

Git was created in 2005 by Linus Torvalds, the creator of the Linux kernel. Git is an open-source project that is licensed under the GNU General Public License (GPL) v2. It was initially made to facilitate kernel development due to the lack of a suitable alternative.

The general workflow for making a Git commit to saving different snapshots goes through the following steps:

Change the content of our project's folder.
Stage or mark the changes we want to save in our next commit.
Commit or save the changes permanently in our project's Git database.

As the third step mentions, Git uses a special database called a repository. This database is kept inside your project's directory under a folder called .git.

Version-Controlling a Project With Git: The Basics

In this section, we'll create a local repository and learn how to manage it using the Git command-line interface (CLI). On macOS and Linux, we can use the default terminal application to follow along with this tutorial.

On Windows, we recommend using Git Bash, which is part of the Git For Windows package. Go to the Git Bash download page, get the installer, run it, and follow the on-screen instruction. Make sure to check the Additional Icons -> On the Desktop to get direct access to Git Bash on your desktop so that you can quickly find and launch the app.

Alternatively, you can also use either Windows' Command Prompt or PowerShell. However, some commands may differ from the commands used in this tutorial.

Initializing a Git Repository

To start version-controlling a project, we need to initialize a new Git repository in the project's root folder or directory. In this tutorial, we'll use a sample project to facilitate the explanation. Go ahead and create a new folder in your file system. Then navigate to that folder in your terminal by running these commands:

shell

$ mkdir sample_project
$ cd sample_project

The first command creates the project's root folder or directory, while the second command allows you to navigate into that folder. Don't close your terminal window. You'll be using it throughout the next sections.

To initialize a Git repository in this folder, we need to use the git init command like in the example below:

shell

$ git init
Initialized empty Git repository in /.../sample_project/.git/

This command creates a subfolder called .git inside the project's folder. The leading dot in the folder's name means that this is a hidden directory. So, you may not see anything on your file manager. You can check the existence of .git with the ls -a, which lists all files in a given folder, including the hidden ones.

Checking the Status of Our Project

Git provides the git status command to allow us to identify the current state of a Git repository. Because our sample_project folder is still empty, running git status will display something like this:

shell

$ git status
On branch main

No commits yet

nothing to commit (create/copy files and use "git add" to track)

When we run git status, we get detailed information about the current state of our Git repository. This command is pretty useful, and we'll turn back to it in multiple moments.

As an example of how useful the git status command is, go ahead and create a file called main.py inside the project's folder using the following commands:

shell

$ touch main.py

$ git status
On branch main

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    main.py

nothing added to commit but untracked files present (use "git add" to track)

With the touch command, we create a new main.py file under our project's folder. Then we run git status again. This time, we get information about the presence of an untracked file called main.py. We also get some basic instructions on how to add this file to our Git repo. Providing these guidelines or instructions is one of the neatest features of git status.

Now, what is all that about untracked files? In the following section, we'll learn more about this topic.

Tracking and Committing Changes

A file in a Git repository can be either tracked or untracked. Any file that wasn't present in the last commit is considered an untracked file. Git doesn't keep a history of changes for untracked files in your project's folder.

In our example, we haven't made any commits to our Git repo, so main.py is naturally untracked. To start tracking it, run the git add command as follows:

shell

$ git add main.py

$ git status
On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
    new file:   main.py

This git add command has added main.py to the list of tracked files. Now it's time to save the file permanently using the git commit command with an appropriate commit message provided with the -m option:

shell

$ git commit -m "Add main.py"
[main (root-commit) 5ac6586] Add main.py
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 main.py

$ git status
On branch master
nothing to commit, working tree clean

We have successfully made our first commit, saving main.py to our Git repository. The git commit command requires a commit message, which we can provide through the -m option. Commit messages should clearly describe what we have changed in our project.

After the commit, our main branch is completely clean, as you can conclude from the git status output.

Now let's start the cycle again by modifying main.py, staging the changes, and creating a new commit. Go ahead and run the following commands:

shell

$ echo "print('Hello, World!')" > main.py
$ cat main.py
print('Hello, World!')

$ git add main.py

$ git commit -m "Create a 'Hello, World!' script  on  main.py"
[main 2f33f7e] Create a 'Hello, World!' script  on  main.py
 1 file changed, 1 insertion(+)

The echo command adds the statement "print('Hello, World!')" to our main.py file. You can confirm this addition with the cat command, which lists the content of one or more target files. You can also open main.py in your favorite editor and update the file there if you prefer.

We can also use the git stage command to stage or add files to a Git repository and include them in our next commit.

We've made two commits to our Git repo. We can list our commit history using the git log command as follows:

shell

$ git log --oneline
2f33f7e (HEAD -> main) Create a 'Hello, World!' script  on  main.py
5ac6586 Add main.py

The git log command allows us to list all our previous commits. In this example, we've used the --oneline option to list commits in a single line each. This command takes us to a dedicated output space. To leave that space, we can press the letter Q on our keyboard.

Using a `.gitignore` File to Skip Unneeded Files

While working with Git, we will often have files and folders that we must not save to our Git repo. For example, most Python projects include a venv/ folder with a virtual environment for that project. Go ahead and create one with the following command:

shell

$ python -m venv venv

Once we've added a Python virtual environment to our project's folder, we can run git status again to check the repo state:

shell

$ git status
On branch main
Untracked files:
  (use "git add <file>..." to include in what will be committed)
    venv/

nothing added to commit but untracked files present (use "git add" to track)

Now the venv/ folder appears as an untracked file in our Git repository. We don't need to keep track of this folder because it's not part of our project's codebase. It's only a tool for working on the project. So, we need to ignore this folder. To do that, we can add the folder to a .gitignore file.

Go ahead and create a .gitignore file in the project's folder. Add the venv/ folders to it and run git status:

shell

$ touch .gitignore
$ echo "venv/" > .gitignore
$ git status
On branch main
Untracked files:
  (use "git add <file>..." to include in what will be committed)
    .gitignore

nothing added to commit but untracked files present (use "git add" to track)

Now git status doesn't list venv/ as an untracked file. This means that Git is ignoring that folder. If we take a look at the output, then we'll see that .gitignore is now listed as an untracked file. We must commit our .gitignore files to the Git repository. This will prevent other developers working with us from having to create their own local .gitignore files.

We can also list multiple files and folders in our .gitignore file one per line. The file even accepts glob patterns to match specific types of files, such as *.txt. If you want to save yourself some work, then you can take advantage of GitHub's gitignore repository, which provides a rich list of predefined .gitignore files for different programming languages and development environments.

We can also set up a global .gitignore file on our computer. This global file will apply to all our Git repositories. If you decide to use this option, then go ahead and create a .gitignore_global in your home folder.

Working With Branches in Git

One of the most powerful features of Git is that it allows us to create multiple branches. A branch is a copy of our project's current status and commits history. Having the option to create and handle branches allows us to make changes to our project without messing up the main line of development.

We'll often find that software projects maintain several independent branches to facilitate the development process. A common branch model distinguishes between four different types of branches:

A main or master branch that holds the main line of development
A develop branch that holds the last developments
One or more feature branches that hold changes intended to add new features
One or more bugfix branches that hold changes intended to fix critical bugs

However, the branching model to use is up to you. In the following sections, we'll learn how to manage branches using Git.

Creating New Branches

Working all the time on the main or master branch isn't a good idea. We can end up creating a mess and breaking the code. So, whenever we want to experiment with a new idea, implement a new feature, fix a bug, or just refactor a piece of code, we should create a new branch.

To kick things off, let's create a new branch called hello on our Git repo under the sample_project folder. To do that, we can use the git branch command followed by the branch's name:

shell

$ git branch hello
$ git branch --list
* main
  hello

The first command creates a new branch in our Git repo. The second command allows us to list all the branches that currently exist in our repository. Again, we can press the letter Q on our keyboard to get back to the terminal prompt.

The star symbol denotes the currently active branch, which is main in the example. We want to work on hello, so we need to activate that branch. In Git's terminology, we need to check out to hello.

Checking Out to a New Branch

Although we have just created a new branch, in order to start working on it, we need to switch to or check out to it by using the git checkout command as follows:

shell

$ git checkout hello
Switched to branch 'hello'

$ git branch --list
  main
* hello

$ git log --oneline
2f33f7e (HEAD -> hello, main) Create a 'Hello, World!' script  on  main.py
5ac6586 Add main.py

The git checkout command takes the name of an existing branch as an argument. Once we run the command, Git takes us to the target branch.

We can derive a new branch from whatever branch we need.

As you can see, git branch --list indicates which branch we are currently on by placing a * symbol in front of the relevant branch name. If we check the commit history with git log --oneline, then we'll get the same as we get from main because hello is a copy of it.

The git checkout can take a -b flag that we can use to create a new branch and immediately check out to it in a single step. That's what most developers use while working with Git repositories. In our example, we could have run git checkout -b hello to create the hello branch and check out to it with one command.

Let's make some changes to our project and create another commit. Go ahead and run the following commands:

shell

$ echo "print('Welcome to PythonGUIs!')" >> main.py
$ cat main.py
print('Hello, World!')
print('Welcome to PythonGUIs!')

$ git add main.py
$ git commit -m "Extend our 'Hello, World' program with a welcome message."
[hello be62476] Extend our 'Hello, World' program with a welcome message.
 1 file changed, 1 insertion(+)

The final command committed our changes to the hello branch. If we compare the commit history of both branches, then we'll see the difference:

shell

$ git log --oneline -1
be62476 (HEAD -> hello) Extend our 'Hello, World' program with a welcome message.

$ git checkout main
Switched to branch 'main'

$ git log --oneline -1
2f33f7e (HEAD -> main) Create a 'Hello, World!' script  on  main.py

In this example, we first run git log --oneline with -1 as an argument. This argument tells Git to give us only the last commit in the active branch's commit history. To inspect the commit history of main, we first need to check out to that branch. Then we can run the same git log command.

Now say that we're happy with the changes we've made to our project in the hello branch, and we want to update main with those changes. How can we do this? We need to merge hello into main.

Merging Two Branches Together

To add the commits we've made in a separate branch back to another branch, we can run what is known as a merge. For example, say we want to merge the new commits in hello into main. In this case, we first need to switch back to main and then run the git merge command using hello as an argument:

shell

$ git checkout main
Already on 'main'

$ git merge hello
Updating 2f33f7e..be62476
Fast-forward
 main.py | 1 +
 1 file changed, 1 insertion(+)

To merge a branch into another branch, we first need to check out the branch we want to update. Then we can run git merge. In the example above, we first check out to main. Once there, we can merge hello.

Deleting Unused Branches

Once we've finished working in a given branch, we can delete the entire branch to keep our repo as clean as possible. Following our example, now that we've merged hello into main, we can remove hello.

To remove a branch from a Git repo, we use the git branch command with the --delete option. To successfully run this command, make sure to switch to another branch before:

shell

$ git checkout main
Already on 'main'

$ git branch --delete hello
Deleted branch hello (was be62476).

$ git branch --list
* main

Deleting unused branches is a good way to keep our Git repositories clean, organized, and up to date. Also, deleting them as soon as we finish the work is even better because having old branches around may be confusing for other developers collaborating with our project. They might end up wondering why these branches are still alive.

Using a GUI Client for Git

In the previous sections, we've learned to use the git command-line tool to manage Git repositories. If you prefer to use GUI tools, then you'll find a bunch of third-party GUI frontends for Git. While they won't completely replace the need for using the command-line tool, they can simplify your day-to-day workflow.

You can get a complete list of standalone GUI clients available on the Git official documentation.

Most popular IDEs and code editors, including PyCharm and Visual Studio Code, come with basic Git integration out-of-the-box. Some developers will prefer this approach as it is directly integrated with their development environment of choice.

If you need something more advanced, then GitKraken is probably a good choice. This tool provides a standalone, cross-platform GUI client for Git that comes with many additional features that can boost your productivity.

Managing a Project With GitHub

If we publish a project on a remote server with support for Git repositories, then anyone with appropriate permissions can clone our project, creating a local copy on their computer. Then, they can make changes to our project, commit them to their local copy, and finally push the changes back to the remote server. This workflow provides a straightforward way to allow other developers to contribute code to your projects.

In the following sections, we'll learn how to create a remote repository on GitHub and then push our existing local repository to it. Before we do that, though, head over to GitHub.com and create an account there if you don't have one yet. Once you have a GitHub account, you can set up the connection to that account so that you can use it with Git.

Setting Up a Secure Connection to GitHub

In order to work with GitHub via the git command, we need to be able to authenticate ourselves. There are a few ways of doing that. However, using SSH is the recommended way. The first step in the process is to generate an SSH key, which you can do with the following command:

shell

$ ssh-keygen -t ed25519 -C "GitHub - name@email.com"

Replace the placeholder email address with the address you've associated with your GitHub account. Once you run this command, you'll get three different prompts in a row. You can respond to them by pressing Enter to accept the default option. Alternatively, you can provide custom responses.

Next, we need to copy the contents of our id_ed25519.pub file. To do this, you can run the following command:

shell

$ cat ~/.ssh/id_ed25519.pub

Select the command's output and copy it. Then go to your GitHub Settings page and click the SSH and GPG keys option. There, select New SSH key, set a descriptive title for the key, make sure that the Key Type is set to Authentication Key, and finally, paste the contents of id_ed25519.pub in the Key field. Finally, click the Add SSH key button.

At this point, you may be asked to provide some kind of Two-Factor Authentication (2FA) code. So, be ready for that extra security step.

Now we can test our connection by running the following command:

shell

$ ssh -T git@github.com
The authenticity of host 'github.com (IP ADDRESS)' can not be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Make sure to check whether the key fingerprint shown on your output matches GitHub's public key fingerprint. If it matches, then enter yes and press Enter to connect. Otherwise, don't connect.

If the connection is successful, we will get a message like this:

shell

Hi USERNAME! You have successfully authenticated, but GitHub does not provide shell access.

Congrats! You've successfully connected to GitHub via SSH using a secure SSH key. Now it's time to start working with GitHub.

Creating and Setting Up a GitHub Repository

Now that you have a GitHub account with a proper SSH connection, let's create a remote repository on GitHub using its web interface. Head over to the GitHub page and click the + icon next to your avatar in the top-right corner. Then select New repository.

Give your new repo a unique name and choose who can see this repository. To continue with our example, we can give this repository the same name as our local project, sample_project.

To avoid conflicts with your existing local repository, don't add .gitignore, README, or LICENSE files to your remote repository.

Next, set the repo's visibility to Private so that no one else can access the code. Finally, click the Create repository button at the end of the page.

If you create a Public repository, make sure also to choose an open-source license for your project to tell people what they can and can't do with your code.

You'll get a Quick setup page as your remote repository has no content yet. Right at the top, you'll have the choice to connect this repository via HTTPS or SSH. Copy the SSH link and run the following command to tell Git where the remote repository is hosted:

shell

$ git remote add origin git@github.com:USERNAME/sample_project.git

This command adds a new remote repository called origin to our local Git repo.

The name origin is commonly used to denote the main remote repository associated with a given project. This is the default name Git uses to identify the main remote repo.

Git allows us to add several remote repositories to a single local one using the git remote add command. This allows us to have several remote copies of your local Git repo.

Pushing a Local Git Repository to GitHub

With a new and empty GitHub repository in place, we can go ahead and push the content of our local repo to its remote copy. To do this, we use the git push command providing the target remote repo and the local branch as arguments:

shell

$ git push -u origin main
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (9/9), 790 bytes | 790.00 KiB/s, done.
Total 9 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:USERNAME/sample_project.git
 * [new branch]      main -> main
branch 'main' set up to track 'origin/main'.

This is the first time we push something to the remote repo sample_project, so we use the -u option to tell Git that we want to set the local main branch to track the remote main branch. The command's output provides a pretty detailed summary of the process.

Note that if you don't add the -u option, then Git will ask what you want to do. A safe workaround is to copy and paste the commands GitHub suggests, so that you don't forget -u.

Using the same command, we can push any local branch to any remote copy of our project's repo. Just change the repo and branch name: git push -u remote_name branch_name.

Now let's head over to our browser and refresh the GitHub page. We will see all of our project files and commit history there.

Now we can continue developing our project and making new commits locally. To push our commits to the remote main branch, we just need to run git push. This time, we don't have to use the remote or branch name because we've already set main to track origin/main.

Pulling Content From a GitHub Repository

We can do basic file editing and make commits within GitHub itself. For example, if we click the main.py file and then click the pencil icon at the top of the file, we can add another line of code and commit those changes to the remote main branch directly on GitHub.

Go ahead and add the statement print("Your Git Tutorial is Here...") to the end of main.py. Then go to the end of the page and click the Commit changes button. This makes a new commit on your remote repository.

This remote commit won't appear in your local commit history. To download it and update your local main branch, use the git pull command:

shell

$ git pull
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), 696 bytes | 174.00 KiB/s, done.
From github.com:USERNAME/sample_project
   be62476..605b6a7  main       -> origin/main
Updating be62476..605b6a7
Fast-forward
 main.py | 1 +
 1 file changed, 1 insertion(+)

Again, the command's output provides all the details about the operation. Note that git pull will download the remote branch and update the local branch in a single step.

If we want to download the remote branch without updating the local one, then we can use the [git fetch](https://git-scm.com/docs/git-fetch) command. This practice gives us the chance to review the changes and commit them to our local repo only if they look right.

For example, go ahead and update the remote copy of main.py by adding another statement like print("Let's go!!"). Commit the changes. Then get back to your local repo and run the following command:

shell

$ git fetch
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), 731 bytes | 243.00 KiB/s, done.
From github.com:USERNAME/sample_project
   605b6a7..ba489df  main       -> origin/main

This command downloaded the latest changes from origin/main to our local repo. Now we can compare the remote copy of main.py to the local copy. To do this, we can use the git diff command as follows:

shell

$ git diff main origin/main
diff --git a/main.py b/main.py
index be2aa66..4f0e7cf 100644
--- a/main.py
+++ b/main.py
@@ -1,3 +1,4 @@
 print('Hello, World!')
 print('Welcome to PythonGUIs!')
 print("Your Git Tutorial is Here...")
+print("Let's go!!")

In the command's output, you can see that the remote branch adds a line containing print("Let's go!!") to the end of main.py. This change looks good, so we can use git pull to commit the change automatically.

Exploring Alternatives to GitHub

While GitHub is the most popular public Git server and collaboration platform in use, it is far from being the only one. GitLab.com and BitBucket are popular commercial alternatives similar to GitHub. While they have paid plans, both offer free plans, with some restrictions, for individual users.

Although, if you would like to use a completely open-source platform instead, Codeberg might be a good option. It's a community-driven alternative with a focus on supporting Free Software. Therefore, in order to use Codeberg, your project needs to use a compatible open-source license.

Optionally, you can also run your own Git server. While you could just use barebones git for this, software such as GitLab Community Edition (CE) and Forgejo provide you with both the benefits of running your own server and the experience of using a service like GitHub.

Conclusion

By now, you're able to use Git for version-controlling your projects. Git is a powerful tool that will make you much more efficient and productive, especially as the scale of your project grows over time.

While this guide introduced you to most of its basic concepts and common commands, Git has many more commands and options that you can use to be even more productive. Now, you know enough to get up to speed with Git.

Working With Classes in Python — Understanding the Intricacies of Python Classes

2023-03-06T06:00:00+00:00

Python supports object-oriented programming (OOP) through classes, which allow you to bundle data and behavior in a single entity. Python classes allow you to quickly model concepts by creating representations of real objects that you can then use to organize your code.

In this tutorial, you'll learn how OOP and classes work in Python. This knowledge will allow you to quickly grasp how you can use their classes and APIs to create robust Python applications.

Defining Classes in Python

Python classes are templates or blueprints that allow us to create objects through instantiation. These objects will contain data representing the object's state, and methods that will act on the data providing the object's behavior.

Instantiation is the process of creating instances of a class by calling the class constructor with appropriate arguments.

Attributes and methods make up what is known as the class interface or API. This interface allows us to operate on the objects without needing to understand their internal implementation and structure.

Alright, it is time to start creating our own classes. We'll start by defining a Color class with minimal functionality. To do that in Python, you'll use the class keyword followed by the class name. Then you provide the class body in the next indentation level:

python

>>> class Color:
...     pass
...

>>> red = Color()

>>> type(red)
<class '__main__.Color'>

In this example, we defined our Color class using the class keyword. This class is empty. It doesn't have attributes or methods. Its body only contains a pass statement, which is Python's way to do nothing.

Even though the class is minimal, it allows us to create instances by calling its constructor, Colo(). So, red is an instance of Color. Now let's make our Color class more fun by adding some attributes.

Adding Class and Instance Аttributes

Python classes allow you to add two types of attributes. You can have class and instance attributes. A class attribute belongs to its containing class. Its data is common to the class and all its instances. To access a class attribute, we can use either the class or any of its instances.

Let's now add a class attribute to our Color class. For example, let's say we need to keep note of how many instance of Color your code creates. Then you can have a color_count attribute:

python

>>> class Color:
...     color_count = 0
...     def __init__(self):
...         Color.color_count += 1
...

>>> red = Color()
>>> green = Color()

>>> Color.color_count
2
>>> red.color_count
2

Now Color has a class attribute called color_count that gets incremented every time we create a new instance. We can quickly access that attribute using either the class directly or one of its instances, like red.

To follow up with this example, say that we want to represent our Color objects using red, green, and blue attributes as part of the RGB color model. These attributes should have specific values for specific instances of the class. So, they should be instance attributes.

To add an instance attribute to a Python class, you must use the .__init__() special method, which we introduced in the previous code but didn't explain. This method works as the instance initializer because it allows you to provide initial values for instance attributes:

python

>>> class Color:
...     color_count = 0
...     def __init__(self, red, green, blue):
...         Color.color_count += 1
...         self.red = red
...         self.green = green
...         self.blue = blue
...

>>> red = Color(255, 0, 0)

>>> red.red
255
>>> red.green
0
>>> red.blue
0

>>> Color.red
Traceback (most recent call last):
    ...
AttributeError: type object 'Color' has no attribute 'red'

Cool! Now our Color class looks more useful. It has the usual class attributes and also three new instance attributes. Note that, unlike class attributes, instance attributes can't be accessed through the class itself. They're specific to a concrete instance.

There's something that jumps into sight in this new version of Color. What is the self argument in the definition of .__init__()? This attribute holds a reference to the current instance. Using the name self to identify the current instance is a strong convention in Python.

We'll use self as the first or even the only argument to instance methods like .__init__(). Inside an instance method, we'll use self to access other methods and attributes defined in the class. To do that, we must prepend self to the name of the target attribute or method instance of the class.

For example, our class has an attribute .red that we can access using the syntax self.red inside the class. This will return the number stored under that name. From outside the class, you need to use a concrete instance instead of self.

Providing Behavior With Methods

A class bundles data (attributes) and behavior (methods) together in an object. You'll use the data to set the object's state and the methods to operate on that data or state.

Methods are just functions that we define inside a class. Like functions, methods can take arguments, return values, and perform different computations on an object's attributes. They allow us to make our objects usable.

In Python, we can define three types of methods in our classes:

Instance methods, which need the instance (self) as their first argument
Class methods, which take the class (cls) as their first argument
Static methods, which take neither the class nor the instance

Let's now talk about instance methods. Say that we need to get the attributes of our Color class as a tuple of numbers. In this case, we can add an .as_tuple() method like the following:

python

class Color:
    representation = "RGB"

    def __init__(self, red, green, blue):
        self.red = red
        self.green = green
        self.blue = blue

    def as_tuple(self):
        return self.red, self.green, self.blue

This new method is pretty straightforward. Since it's an instance method, it takes self as its first argument. Then it returns a tuple containing the attributes .red, .green, and .blue. Note how you need to use self to access the attributes of the current instance inside the class.

This method may be useful if you need to iterate over the RGB components of your color objects:

python

>>> red = Color(255, 0, 0)
>>> red.as_tuple()
(255, 0, 0)

>>> for level in red.as_tuple():
...     print(level)
...
255
0
0

Our as_tuple() method works great! It returns a tuple containing the RGB components of our color objects.

We can also add class methods to our Python classes. To do this, we need to use the @classmethod decorator as follows:

python

class Color:
    representation = "RGB"

    def __init__(self, red, green, blue):
        self.red = red
        self.green = green
        self.blue = blue

    def as_tuple(self):
        return self.red, self.green, self.blue

    @classmethod
    def from_tuple(cls, rbg):
        return cls(*rbg)

The from_tuple() method takes a tuple object containing the RGB components of a desired color as an argument, creates a valid color object from it, and returns the object back to the caller:

python

>>> blue = Color.from_tuple((0, 0, 255))
>>> blue.as_tuple()
(0, 0, 255)

In this example, we use the Color class to access the class method from_tuple(). We can also access the method using a concrete instance of this class. However, in both cases, we'll get a completely new object.

Finally, Python classes can also have static methods that we can define with the @staticmethod decorator:

python

class Color:
    representation = "RGB"

    def __init__(self, red, green, blue):
        self.red = red
        self.green = green
        self.blue = blue

    def as_tuple(self):
        return self.red, self.green, self.blue

    @classmethod
    def from_tuple(cls, rbg):
        return cls(*rbg)

    @staticmethod
    def color_says(message):
        print(message)

Static methods don't operate either on the current instance self or the current class cls. These methods can work as independent functions. However, we typically put them inside a class when they are related to the class, and we need to have them accessible from the class and its instances.

Here's how the method works:

python

>>> Color.color_says("Hello from the Color class!")
Hello from the Color class!

>>> red = Color(255, 0, 0)
>>> red.color_says("Hello from the red instance!")
Hello from the red instance!

This method accepts a message and prints it on your screen. It works independently from the class or instance attributes. Note that you can call the method using the class or any of its instances.

Writing Getter & Setter Methods

Programming languages like Java and C++ rely heavily on setter and getter methods to retrieve and update the attributes of a class and its instances. These methods encapsulate an attribute allowing us to get and change its value without directly accessing the attribute itself.

For example, say that we have a Label class with a text attribute. We can make text a non-public attribute and provide getter and setter methods to manipulate the attributes according to our needs:

python

class Label:
    def __init__(self, text):
        self.set_text(text)

    def text(self):
        return self._text

    def set_text(self, value):
        self._text = str(value)

In this class, the text() method is the getter associated with the ._text attribute, while the set_text() method is the setter for ._text. Note how ._text is a non-public attribute. We know this because it has a leading underscore on its name.

The setter method calls str() to convert any input value into a string. Therefore, we can call this method with any type of object. It will convert any input argument into a string, as you will see in a moment.

If you come from programming languages like Java or C++, you need to know Python doesn't have the notion of private, protected, and public attributes. In Python, you'll use a naming convention to signal that an attribute is non-public. This convention consists of adding a leading underscore to the attribute's name. Note that this naming pattern only indicates that the attribute isn't intended to be used directly. It doesn't prevent direct access, though.

This class works as follows:

python

>>> label = Label("Python!")

>>> label.text()
'Python!'

>>> label.set_text("Classes!")
>>> label.text()
'Classes!'

>>> label.set_text(123)
>>> label.text()
'123'

In this example, we create an instance of Label. The original text is passed to the class constructor, Label(), which automatically calls __init__() to set the value of ._text by calling the setter method text(). You can use text() to access the label's text and set_text() to update it. Remember that any input will be converted into a string, as we can see in the final example above.

The getter and setter pattern is pretty common in languages like Java and C++. However, this pattern is less popular among Python developers. Instead, they use the @property decorator to hide attributes behind properties.

Here's how most Python developer will write their Label class:

python

class Label:
    def __init__(self, text):
        self.text = text

    @property
    def text(self):
        return self._text

    @text.setter
    def text(self, value):
        self._text = str(value)

This class defines .text as a property. This property has getter and setter methods. Python calls them automatically when we access the attribute or update its value in an assignment:

python

>>> label = Label("Python!")

>>> label.text
'Python!'

>>> label.text = "Class"
>>> label.text
'Class'

>>> label.text = 123
>>> label.text
'123'

Python properties allow you to add function behavior to your attributes while permitting you to use them as normal attributes instead of as methods.

Writing Special Methods

Python supports many special methods, also known as dunder or magic methods, that are part of its class mechanism. We can identify these methods because their names start and end with a double underscore, which is the origin of their other name: dunder methods.

These methods accomplish different tasks in Python's class mechanism. They all have a common feature: Python calls them automatically depending on the operation we run.

For example, all Python objects are printable. We can print them to the screen using the print() function. Calling print() internally falls back to calling the target object's __str__() special method:

python

>>> label = Label("Python!")

>>> print(label)
<__main__.Label object at 0x10354efd0>

In this example, we've printed our label object. This action provides some information about the object and the memory address where it lives. However, the actual output is not very useful from the user's perspective.

Fortunately, we can improve this by providing our Label class with an appropriate __str__() method:

python

class Label:
    def __init__(self, text):
        self.text = text

    @property
    def text(self):
        return self._text

    @text.setter
    def text(self, value):
        self._text = str(value)

    def __str__(self):
        return self.text

The __str__() method must return a user-friendly string representation for our objects. In this case, when we print an instance of Label to the screen, the label's text will be displayed:

python

>>> label = Label("Python!")

>>> print(label)
Python!

As you can see, Python takes care of calling __str__() automatically when we use the print() function to display our instances of Label.

Another special method that belongs to Python's class mechanism is __repr__(). This method returns a developer-friendly string representation of a given object. Here, developer-friendly implies that the representation should allow a developer to recreate the object itself.

python

class Label:
    def __init__(self, text):
        self.text = text

    @property
    def text(self):
        return self._text

    @text.setter
    def text(self, value):
        self._text = str(value)

    def __str__(self):
        return self.text

    def __repr__(self):
        return f"{type(self).__name__}(text='{self.text}')"

The __repr__() method returns a string representation of the current objects. This string differs from what __str__() returns:

python

>>> label = Label("Python!")
>>> label
Label(text='Python!')

Now when you access the instance on your REPL session, you get a string representation of the current object. You can copy and paste this representation to recreate the object in an appropriate environment.

Reusing Code With Inheritance

Inheritance is an advanced topic in object-oriented programming. It allows you to create hierarchies of classes where each subclass inherits all the attributes and behaviors from its parent class or classes. Arguably, code reuse is the primary use case of inheritance.

Yes, we code a base class with a given functionality and make that functionality available to its subclass through inheritance. This way, we implement the functionality only once and reuse it in every subclass.

Python classes support single and multiple inheritance. For example, let's say we need to create a button class. This class needs .width and .height attributes that define its rectangular shape. The class also needs a label for displaying some informative text.

We can code this class from scratch, or we can use inheritance and reuse the code of our current Label class. Here's how to do this:

python

class Button(Label):
    def __init__(self, text, width, height):
        super().__init__(text)
        self.width = width
        self.height = height

    def __repr__(self):
        return (
            f"{type(self).__name__}"
            f"(text='{self.text}', "
            f"width={self.width}, "
            f"height={self.height})"
        )

To inherit from a parent class in Python, we need to list the parent class or classes in the subclass definition. To do this, we use a pair of parentheses and a comma-separated list of parent classes. If we use several parent classes, then we're using multiple inheritance, which can be challenging to reason about.

The first line in __init__() calls the __init__() method on the parent class to properly initialize its .text attribute. To do this, we use the built-in super() function. Then we define the .width and .height attributes, which are specific to our Button class. Finally, we provide a custom implementation of __repr__().

Here's how our Button class works:

python

>>> button = Button("Ok", 10, 5)

>>> button.text
'Ok'
>>> button.text = "Click Me!"
>>> button.text
'Click Me!'

>>> button.width
10
>>> button.height
5

>>> button
Button(text='Ok', width=10, height=5)
>>> print(button)
Click Me!

As you can conclude from this code, Button has inherited the .text attribute from Label. This attribute is completely functional. Our class has also inherited the __str__() method from Label. That's why we get the button's text when we print the instance.

As we've seen, Python allows us to write classes that work as templates that you can use to create concrete objects that bundle together data and behavior. The building blocks of Python classes are:

Attributes, which hold the data in a class
Methods, which provide the behaviors of a class

The attributes of a class define the class's data, while the methods provide the class's behaviors, which typically act on that data.

To better understand OOP and classes in Python, we should first discuss some terms that are commonly used in this aspect of Python development:

Classes are blueprints or templates for creating objects -- just like a blueprint for creating a car, plane, house, or anything else. In programming, this blueprint will define the data (attributes) and behavior (methods) of the object and will allow us to create multiple objects of the same kind.
Objects or Instances are the realizations of a class. We can create objects from the blueprint provided by the class. For example, you can create John's car from a Car class.
Methods are functions defined within a class. They provide the behavior of an object of that class. For example, our Car class can have methods to start the engine, turn right and left, stop, and so on.
Attributes are properties of an object or class. We can think of attributes as variables defined in a class or object. Therefore, we can have:
- class attributes, which are specific to a concrete class and common to all the instances of that class. You can access them either through the class or an object of that class. For example, if we're dealing with a single car manufacturer, then our Car class can have a manufacturer attribute that identifies it.
- instance attributes, which are specific to a concrete instance. You can access them through the specific instance. For example, our Car class can have attributes to store properties such as the maximum speed, the number of passengers, the car's weight, and so on.
Instantiation is the process of creating an individual instance from a class. For example, we can create John's car, Jane's car, and Linda's car from our Car class through instantiation. In Python, this process runs through two steps:
1. Instance creation: Creates a new object and allocates memory for storing it.
2. Instance initialization: Initializes all the attributes of the current object with appropriate values.
Inheritance is a mechanism of code reuse that allows us to inherit attributes and methods from one or multiple existing classes. In this context, we'll hear terms like:
- Parent class: The class we're inheriting from. This class is also known as the superclass or base class. If we have one parent class, then we're using single inheritance. If we have more than one parent class, then we're using multiple inheritance.
- Child class: The class that inherits from a given parent. This class is also known as the subclass.

Don't feel frustrated or bad if you don't understand all these terms immediately. They'll become more familiar with use as you use them in your own Python code.

Conclusion

Now you know the basics of Python classes. You also learned fundamental concepts of object-oriented programming, such as inheritance.

Getting started with VS Code for Python — Setting up a Development Environment for Python programming

2022-09-21T09:00:00+00:00

Setting up a working development environment is the first step for any project. Your development environment setup will determine how easy it is to develop and maintain your projects over time. That makes it important to choose the right tools for your project. This article will guide you through how to set up Visual Studio Code, which is a popular free-to-use, cross-platform code editor developed by Microsoft, in order to develop Python applications.

Visual Studio Code is not to be confused with Visual Studio, which is a separate product also offered by Microsoft. Visual Studio is a fully-fledged IDE that is mainly geared towards Windows application development using C# and the .NET Framework.

Setup a Python environment

In case you haven't already done this, Python needs to be installed on the development machine. You can do this by going to python.org and grabbing the specific installer for either Windows or macOS. Python is also available for installation via Microsoft Store on Windows devices.

Make sure that you select the option to Add Python to PATH during installation (via the installer).

If you are on Linux, you can check if Python is already installed on your machine by typing python3 --version in a terminal. If it returns an error, you need to install it from your distribution's repository. On Ubuntu/Debian, this can be done by typing sudo apt install python3. Both pip (or pip3) and venv are distributed as separate packages on Ubuntu/Debian and can also be installed by typing sudo apt install python3-pip python3-venv.

Setup Visual Studio Code

First, head over to to code.visualstudio.com and grab the installer for your specific platform.

If you are on a Raspberry Pi (with Raspberry Pi OS), you can also install VS Code by simply typing sudo apt install code. On Linux distributions that support Snaps, you can do it by typing sudo snap install code --classic.

Once VS Code is installed, head over to the Extensions tab in the sidebar on the left by clicking on it or by pressing CTRL+SHIFT+X. Search for the 'Python' extension published by Microsoft and click on Install.

The Extensions tab in the left-hand sidebar.

Usage and Configuration

Now that you have finished setting up VS Code, you can go ahead and create a new Python file. Remember that the Python extension only works if you open a .py file or have selected the language mode for the active file as Python.

To change the language mode for the active file, simply press CTRL+K once and then press M after releasing the previous keys. This kind of keyboard shortcut is called a chord in VS Code. You can see more of them by pressing CTRL+K CTRL+S (another chord).

The Python extension in VS Code allows you to directly run a Python file by clicking on the 'Play' button on the top-right corner of the editor (without having to type python file.py in the terminal).

You can also do it by pressing CTRL+SHIFT+P to open the Command Palette and running the > Python: Run File in Terminal command.

Finally, you can configure VS Code's settings by going to File > Preferences > Settings or by pressing CTRL+COMMA. In VS Code, each individual setting has an unique identifier which you can see by clicking on the cog wheel that appears to the left of each setting and clicking on 'Copy Setting ID'. This ID is what will be referred to while talking about a specific setting. You can also search for this ID in the search bar under Settings.

Linting and Formatting Support (Optional)

Linters make it easier to find errors and check the quality of your code. On the other hand, code formatters help keep the source code of your application compliant with PEP (Python Enhancement Proposal) standards, which make it easier for other developers to read your code and collaborate with you.

For VS Code to provide linting support for your projects, you must first install a preferred linter like flake8 or pylint.

bash

pip install flake8

Then, go to Settings in VS Code and toggle the relevant setting (e.g. python.linting.flake8Enabled) for the Python extension depending on what you installed. You also need to make sure that python.linting.enabled is toggled on.

A similar process must be followed for code formatting. First, install something like autopep8 or black.

bash

pip install autopep8

You then need to tell VS Code which formatter to use by modifying python.formatting.provider and toggle on editor.formatOnSave so that it works without manual intervention.

If pip warns that the installed modules aren't in your PATH, you may have to specify the path to their location in VS Code (under Settings). Follow the method described under Working With Virtual Environments to do that.

Now, when you create a new Python file, VS Code automatically gives you a list of Problems (CTRL+SHIFT+M) in your program and formats the code on saving the file.

Identified problems in the source code, along with a description and line/column numbers.

You can also find the location of identified problems from the source overview on the right hand, inside the scrollbar.

Working With Virtual Environments

Virtual environments are a way of life for Python developers. Most Python projects require the installation of external packages and modules (via pip). Virtual environments allow you to separate one project's packages from your other projects, which may require a different version of those same packages. Hence, it allows all those projects to have the specific dependencies they require to work.

The Python extension makes it easier for you by automatically activating the desired virtual environment for the in-built terminal and Run Python File command after you set the path to the Python interpreter. By default, the path is set to use the system's Python installation (without a virtual environment).

To use a virtual environment for your project/workspace, you need to first make a new one by opening a terminal (View > Terminal) and typing python -m venv .venv. Then, you can set the default interpreter for that project by opening the Command Palette (CTRL+SHIFT+P) and selecting > Python: Select Interpreter.

You should now either close the terminal pane in VS Code and open a new one or type source .venv/bin/activate into the existing one to start using the virtual environment. Then, install the required packages for your project by typing pip install <package_name>.

VS Code, by default, looks for tools like linters and code formatters in the current Python environment. If you don't want to keep installing them over and over again for each new virtual environment you make (unless your project requires a specific version of that tool), you can specify the path to their location under Settings in VS Code. - flake8 - python.linting.flake8Path - autopep8 - python.formatting.autopep8Path

To find the global location of these packages on macOS and Linux, type which flake8 and which autopep8 in a terminal. If you are on Windows, you can use where <command_name>. Both these commands assume that flake8 and autopep8 are in your PATH.

Understanding Workspaces in VS Code

VS Code has a concept of Workspaces. Each 'project folder' (or the root/top folder) is treated as a separate workspace. This allows you to have project-specific settings and enable/disable certain extensions for that workspace. It is also what allows VS Code to quickly recover the UI state (e.g. files that were previously kept open) when you open that workspace again.

In VS Code, each workspace (or folder) has to be 'trusted' before certain features like linters, autocomplete suggestions and the in-built terminal are allowed to work.

In the context of Python projects, if you tend to keep your virtual environments outside the workspace (where VS Code is unable to detect it), you can use this feature to set the default path to the Python interpreter for that workspace. To do that, first Open a Folder (CTRL+K CTRL+O) and then go to File > Preferences > Settings > Workspace to modify python.defaultInterpreterPath.

Setting the default interpreter path for the workspace.

In VS Code settings you can search for settings by name using the bar at the top.

You can also use this approach to do things like use a different linter for that workspace or disable the code formatter for it. The workspace-specific settings you change are saved in a .vscode folder inside that workspace, which you can share with others.

If your VS Code is not recognizing libraries you are using in your code, double check the correct interpreter is being used. You can find which Python version you're using on the command line by running which python or which python3 on macOS/Linux, or where python or where python3 on Windows.

Working With Git in VS Code (Optional)

Using Version Control is required for developing applications. VS Code does have in-built support for Git but it is pretty barebones, not allowing much more than tracking changes that you have currently made and committing/pushing those changes once you are done.

For the best experience, it is recommended to use the GitLens extension. It lets you view your commit history, check who made the changes and much more. To set it up, you first need to have Git set up on your machine (go here) and then install GitLens from the Extensions tab in the sidebar on the left. You can now use those Git-related features by going to the Git tab in the sidebar (CTRL+SHIFT+G).

There are more Git-related extensions you could try as well, like Git History and GitLab Workflow. Give them a whirl too!

Community-driven & open source alternatives

While VS Code is open source (MIT-licensed), the distributed versions include some Microsoft-specific proprietary modifications, such as telemetry (app tracking). If you would like to avoid this, there is also a community-driven distribution of Visual Studio Code called VSCodium that provides freely-licensed binaries without telemetry.

Due to legal restrictions, VSCodium is unable to use the official Visual Studio Marketplace for extensions. Instead, it uses a separate vendor neutral, open source marketplace called Open VSX Registry. It doesn't have every extension, especially proprietary ones, and some are not kept up-to-date but both the Python and GitLens extensions are available on it.

You can also use the open source Jedi language server for the Python extension, rather than the bundled Pylance language server/extension, by configuring the python.languageServer setting. You can then completely disable Pylance by going to the Extensions tab. Note that, if you are on VSCodium, Jedi is used by default (as Pylance is not available on Open VSX Registry) when you install the Python extension.

Conclusion

Having the right tools and making sure they're set up correctly will greatly simplify your development process. While Visual Studio starts as a simple tool, it is flexible and extendable with plugins to suit your own preferred workflow. In this tutorial we've covered the basics of setting up your environment, and you should now be ready to start developing your own applications with Python!

For an in-depth guide to building Python GUIs with PyQt6 see my book, Create GUI Applications with Python & Qt6.

Using MicroPython and uploading libraries on Raspberry Pi Pico — Using rshell to upload custom code

2021-02-22T08:00:00+00:00

MicroPython is an implementation of the Python 3 programming language, optimized to run microcontrollers. It's one of the options available for programming your Raspberry Pi Pico and a nice friendly way to get started with microcontrollers.

MicroPython can be installed easily on your Pico, by following the instructions on the Raspberry Pi website (click the Getting Started with MicroPython tab and follow the instructions).

After that point you might get a bit stuck. The Pico documentation covers connecting to the Pico from a Pi, so if you're wanting to code from your own computer you'll need something else. One option is the Thonny IDE which you use to write and upload code to your Pico. It's got a nice friendly interface for working with Python.

But if you don't want to change your IDE, or want a way to communicate with your Pico from the command line? You're in luck: there is a simple tool for accessing the MicroPython REPL on your Pico and uploading custom Python scripts or libraries you may wish to use: rshell

In this tutorial I'll take you through working with MicroPython in rshell, coding live and uploading custom scripts to your Pico.

Installing rshell

Rshell itself is built on Python (not MicroPython) and can be installed and run locally on your main machine. You can install it like any other Python library.

bash

python -m pip install rshell

Unfortunately, the current version of rshell does not always play nicely with the Raspberry Pico. If you have problems you can install a fixed version in the pico branch from the rshell repository. You can install this directly from Github with the following --

bash

python -m pip install https://github.com/dhylands/rshell/archive/pico.zip

This will download the latest version of the pico branch (as a .zip) and install this in your Python environment.

Once installed, you will have access to a new command line tool rshell.

The rshell interface

To use rshell from the command line, enter rshell at your command prompt. You will see a welcome message and the prompt will turn green, to indicate you're in rshell mode.

The rshell interface on Windows 10

The rshell interface on macOS

If previous pip install worked but the rshell command doesn't work, then you may have a problem with your Python paths.

To see the commands available in rshell, enter help and press Enter.

python

help

Documented commands (type help <topic>):
========================================
args    cat  connect  date  edit  filesize  help  mkdir  rm     shell
boards  cd   cp       echo  exit  filetype  ls    repl   rsync

Use the exit command to exit rshell.

You can exit rshell at any time by entering exit or pressing Ctrl-C. Once exited the prompt will turn white.

The basic file operation commands are shown below.

cd <dirname> change directory
cp <from> <to> copy a file
ls list current directory
rm <filename> remove (delete) a file
filesize <filename> give the size of a file in bytes

If you type ls and press enter you will see a listing of your current folder on your host computer. The same goes for any of the other file operations, until we've connect a board and opened it's file storage -- so be careful! We'll look at how to connect to a MicroPython board and work with the files on it next.

Connecting to your Pico with rshell

Enter boards to see a list of MicroPython boards connected to your computer. If you don't have any connected boards, you'll see the message No boards connected.

If your board isn't connected plug your Pico in now. You can use the connect command to connect to the board, but for that you'll need to know which port it is on. Save yourself some effort and just restart rshell to connect automatically. To do this, type exit and press Enter (or press Ctrl-C) to exit and then restart rshell by entering rshell again at the prompt.

If a board is connected when you start rshell you will see something like the following...

python

C:\Users\Gebruiker>rshell
Connecting to COM4 (buffer-size 128)...
Trying to connect to REPL  connected

Or an equivalent on macOS...

python

Martins-Mac: ~ mfitzp$ rshell
Connecting to /dev/cu.usbmodem0000000000001 (buffer-size 128)
Trying to connect to REPL  connected

...which shows you've connected to the MicroPython REPL on the Raspberry Pi Pico. Once connected the boards command will return some information about the connected board, like the following.

python

pyboard @ COM4 connected Epoch: 1970 Dirs:

The name on the left is the type of board (Pico appears as pyboard) and connected port (here COM4). The label at the end Dirs: will list any files on the Pico -- currently none.

Starting a REPL

With the board connected, you can enter the Pico's REPL by entering the repl command. This will return something like the following

python

repl
Entering REPL. Use Control-X to exit.
>
MicroPython v1.14 on 2021-02-14; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>>
>>>

You are now writing Python on the Pico! Try entering print("Hello!") at the REPL prompt.

python

MicroPython v1.14 on 2021-02-14; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>>
>>> print("Hello!")
Hello!

As you can see, MicroPython works just like normal Python. If you enter help() and press Enter, you'll get some basic help information about MicroPython on the Pico. Helpfully, you also get a small reference to how the pins on the Pico are numbered and the different ways you have to control them.

python

Type "help()" for more information.
>>> help()
Welcome to MicroPython!

For online help please visit https://micropython.org/help/.

For access to the hardware use the 'machine' module.  RP2 specific commands
are in the 'rp2' module.

Quick overview of some objects:
  machine.Pin(pin) -- get a pin, eg machine.Pin(0)
  machine.Pin(pin, m, [p]) -- get a pin and configure it for IO mode m, pull mode p
    methods: init(..), value([v]), high(), low(), irq(handler)
  machine.ADC(pin) -- make an analog object from a pin
    methods: read_u16()
  machine.PWM(pin) -- make a PWM object from a pin
    methods: deinit(), freq([f]), duty_u16([d]), duty_ns([d])
  machine.I2C(id) -- create an I2C object (id=0,1)
    methods: readfrom(addr, buf, stop=True), writeto(addr, buf, stop=True)
             readfrom_mem(addr, memaddr, arg), writeto_mem(addr, memaddr, arg)
  machine.SPI(id, baudrate=1000000) -- create an SPI object (id=0,1)
    methods: read(nbytes, write=0x00), write(buf), write_readinto(wr_buf, rd_buf)
  machine.Timer(freq, callback) -- create a software timer object
    eg: machine.Timer(freq=1, callback=lambda t:print(t))

Pins are numbered 0-29, and 26-29 have ADC capabilities
Pin IO modes are: Pin.IN, Pin.OUT, Pin.ALT
Pin pull modes are: Pin.PULL_UP, Pin.PULL_DOWN

Useful control commands:
  CTRL-C -- interrupt a running program
  CTRL-D -- on a blank line, do a soft reset of the board
  CTRL-E -- on a blank line, enter paste mode

For further help on a specific object, type help(obj)
For a list of available modules, type help('modules')

You can run help() in the REPL any time you need a reminder.

While we're here, lets flash the LED on the Pico board.

Enter the following at the REPL prompt...

python

from machine import Pin
led = Pin(25, Pin.OUT)

led.toggle()

Every time you call led.toggle() the LED will toggle from ON to OFF or OFF to ON.

To exit the REPL at any time press Ctrl-X

Uploading a file

MicroPython comes with a lot of built-in support for simple devices and communication protocols -- enough to build some quite fun things just by hacking in the REPL. But there are also a lot of libraries available for working with more complex hardware. To use these, you need to be able to upload them to your Pico! Once you can upload files, you can also edit your own code locally on your own computer and upload it from there.

To keep things simple, lets create our own "library" that adjusts the brightness of the LED on the Pico board -- exciting I know. This library contains a single function ledon which accepts a single parameter brightness between 0 and 65535.

python

from machine import Pin, PWM

led = PWM(Pin(25))

def ledon(brightness=65535):
    led.duty_u16(brightness)

Don't worry if you don't understand it, we'll cover how this works later. The important bit now is getting this on your Pico.

Take the code above and save it in a file named picoled.py on your main computer, in the same folder you're executing rshell from. We'll upload this file to the Pico next.

Start rshell if you are not already in it -- look for the green prompt. Enter boards at the prompt to get a list of connected boards.

bash

pyboard @ COM4 connected Epoch: 1970 Dirs:

To see the directory contents of the pyboard device, you can enter:

bash

ls /pyboard

You should see nothing listed. The path /pyboard works like a virtual folder meaning you can copy files to this location to have the uploaded to your Pico. It is only available by a pyboard is connected. To upload a file, we copy it to this location. Enter the following at the prompt.

bash

cp picoled.py /pyboard/picoled.py

After you press Enter you'll see a message confirming the copy is taking place

python

C:\Users\Gebruiker> cp picoled.py /pyboard
Copying 'C:\Users\Gebruiker/picoled.py' to '/pyboard/picoled.py' ...

Once the copy is complete, run boards again at the prompt and you'll see the file listed after the Dirs: section, showing that it's on the board.

bash

C:\Users\Gebruiker> boards
pyboard @ COM4 connected Epoch: 1970 Dirs: /picoled.py /pyboard/picoled.py

You can also enter ls /pyboard to see the listing directly.

bash

C:\Users\Gebruiker> ls /pyboard
picoled.py

If you ever need to upload multiple files, just repeat the upload steps until everything is where it needs to be. You can always drop in and out of the REPL to make sure things work.

Using uploaded libraries

Now we've uploaded our library, we can use it from the REPL. To get to the MicroPython REPL enter the repl command in rshell as before. To use the library we uploaded, we can import it, just like any other Python library.

python

MicroPython v1.14 on 2021-02-14; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>>
>>> import picoled
>>> picoled.ledon(65535)
>>> picoled.ledon(30000)
>>> picoled.ledon(20000)
>>> picoled.ledon(10000)

Or to pulse the brightness of the LED...

python

>>> import picoled
>>> import time
>>> while True:
...     for a in range(0, 65536, 10000):
...         picoled.ledon(a)
...         time.sleep(0.1)

Auto-running Python

So far we've been uploading code and running it manually, but once you start building projects you'll want your code to run automatically.

When it starts, MicroPython runs two scripts by default: boot.py and main.py, in that order. By uploading your own script with the name main.py it will run automatically every time the Raspberry Pico starts.

Let's update our "library" to become an auto script that runs at startup. Save the following code to a script named main.py.

python

from machine import Pin, PWM
from time import sleep

led = PWM(Pin(25))

def ledon(brightness=65535):
    led.duty_u16(brightness)


while True:
    for a in range(0, 65536, 10000):
        ledon(a)
        sleep(0.1)

In rshell run the command to copy the file to main.py on the board.

bash

cp main.py /pyboard/main.py

Don't copy this file to boot.py -- the loop will block the REPL startup and you won't be able to connect to your Pico to delete it again! If you do this, use the Resetting Flash memory instructions to clear your Pico. You will need to re-install MicroPython afterwards.

Once the main.py file is uploaded, restart your Pico -- either unplug and re-plug it, or press Ctrl-D in the REPL -- and the LED will start pulsing automatically. The script will continue running until it finishes, or the Pico is reset. You can replace the main.py script at any time to change the behavior, or delete it with.

bash

rm /pyboard/main.py

What's next?

Now you can upload libraries to your Pico you can get experimenting with the many MicroPython libraries that are available.

If you're looking for some more things to do with MicroPython on your Pico, there are some MicroPython examples available from Raspberry Pi themselves, and also the MicroPython documentation for language/API references.

3-axis Accelerometer-Gyro — Measuring acceleration and orientation with an MPU6050

2019-01-01T08:00:00+00:00

Measuring acceleration and rotation has a lot of useful applications, from drone or rocket stablisation to making physically interactive handheld games.

An accelerometer measures proper acceleration, meaning the rate of change of velocity relative to it's own rest frame. This is in contrast to coordinate acceleration, which is relative to a fixed coordinate system. The practical upshot of this is that at rest on Earth an accelerometer will measure acceleration due to the Earth's gravity, of g ≈ 9.81 m/s. An accelerometer in freefall will measure zero. This can be adjusted for with calibration.

A gyroscope (from Ancient Greek γῦρος "circle" and σκοπέω "to look") in contrast measures orientation and angular velocity, or rotation around an an axis. Angular velocity will always be zero at rest.

The availability of cheap single-chip accelerometer-gyroscope packages makes them practical for any project.

MPU6050

The MPU6050 is a nifty little 3-axis accelerometer and gyro package, providing measurements for acceleration along and rotation around 3 axes. It also contains an inbuilt temperature sensor. There are 4 configurable ranges for the gyro and accelerometer, meaning it can be used for both micro and macro measurements. Communication is via a simple I2C interface.

Gyro Full Scale Range (°/sec)	Gyro Sensitivity (LSB/°/sec)	Gyro Rate Noise (dps/√Hz)	Accel Full Scale Range (g)	Accel Sensitivity (LSB/g)
±250	131	0.005	±2	16384
±500	65.5	0.005	±4	8192
±1000	32.8	0.005	±8	4096
±2000	16.4	0.005	±16	2048

See the full MPU-6050 Product Specificiation.

Requirements
Wemos D1 v2.2+ or good imitations.	amazon
3-axis Gyroscope Based on MPU6050 chip	amazon
Breadboard Any size will do.	amazon
Wires Loose ends, or jumper leads.

Setting up

The MPU-6050 provides an I2C interface for communication. There are Python libraries available which simplify the communication further and return the measurements in a simple format. The examples here are using this MPU6050 library.

You can download the mpu6050.py file directly. Click Raw format and save the file with a .py extension. Upload the file to your device using ampy tool (or the WebREPL):

python

ampy --port /dev/tty.wchusbserial141120 put mpu6050.py

With the mpu6050.py file on your Wemos D1, you can import it as any other Python module. Connect to your device, and then in the REPL enter:

python

from machine import I2C, Pin
import mpu6050

If the import mpu6050 succeeds, the package is correctly uploaded and you're good to go.

Wire up the MPU6050, connecting pins D1 to SCL and D2 to SDA. Provide power from G and 5V. The light on the MPU6050 should light up once it's active.

Reading values

With the mpu6050 Python library on your device, and the MPU6050 module wired up, you can connect to the shell and start talking to it.

python

from machine import I2C, Pin
import mpu6050

i2c = I2C(scl=Pin(5), sda=Pin(4))
accel = mpu6050.accel(i2c)

With the accel object, we can read values from the sensor with .get_values(). This will return a dictionary of measurements from the sensor.

python

>>> accel.get_values()
{'GyZ': -46, 'GyY': -135, 'GyX': -1942, 'Tmp': 26.7888, 'AcZ': 24144, 'AcY': 68, 'AcX': -1004}

There are 3 sets of measurements returned from the sensor — acceleration, rotation (gyration) and temperature. The acceleration and rotation measurements provide 3 values each, one for each of the axes (X, Y, Z).

Measurement	Description
AcX	Acceleration along X axis
AcY	Acceleration along Y axis
AcZ	Acceleration along Z axis
GyX	Rotation around X axis
GyY	Rotation around Y axis
GyZ	Rotation around Z axis
Tmp	Temperature °C

The direction of the X and Y axes relative to the sensor are shown on the module itself. But you can always just adjust your code by trial and error.

Smoothing

If you repeatedly read measurements from the sensor in this way you'll notice that they're bouncing all over the place. This is normal for analog sensors. Before we can use the measurements from the sensor, we need to smooth out these random fluctuations to leave us with real representative data.

A simple way to do this is to read multiple values and take the mean (or median) of all the values. The sensor returns multiple values, so we need to average all of these individually.

python

def get_smoothed_values(n_samples=10):
    """
    Get smoothed values from the sensor by sampling
    the sensor `n_samples` times and returning the mean.
    """
    result = {}
    for _ in range(n_samples):
        data = accel.get_values()

        for k in data.keys():
            # Add on value / n_samples (to generate an average)
            # with default of 0 for first loop.
            result[k] = result.get(k, 0) + (data[k] / n_samples)

    return result

Running the above function will sample the sensor ten times, and return the averaged values.

python

>>> accel.get_values()
{'GyZ': -46, 'GyY': -135, 'GyX': -1942, 'Tmp': 26.7888, 'AcZ': 24144, 'AcY': 68, 'AcX': -1004}

If you repeatedly run this you should find the samples have stabilised quite a bit. As the values are in the range ±32768 the above measurements are actually pretty close to zero, though Z acceleration is notably higher. AcZ is the acceleration measurement in the Z (straight-up) axis and this value is the acceleration due to gravity of g ≈ 9.81 m/s.

The measured value 24144/16384 (at lowest measurement sensitivity) gives 1.47g so it's still off a bit.

The other offsets at rest are just inherent errors in the chip (individual, not the model), they're not interesting. To get our measurements centred around zero we can must identify this bias and adjust for it through calibration.

Calibration

If we take a number of repeated sensor measurements over time we can determine the standard, or average, deviation from zero over time. This offset can then be subtracted from future measurements to correct them. The device must be at rest and not changing for this to work reliably.

python

def calibrate(threshold=50, n_samples=100):
    """
    Get calibration date for the sensor, by repeatedly measuring
    while the sensor is stable. The resulting calibration
    dictionary contains offsets for this sensor in its
    current position.
    """
    while True:
        v1 = get_accel(n_samples)
        v2 = get_accel(n_samples)
        # Check all consecutive measurements are within
        # the threshold. We use abs() so all calculated
        # differences are positive.
        if all(abs(v1[k] - v2[k]) < threshold for k in v1.keys()):
            return v1  # Calibrated.

The all(abs(v1[k] - v2[k]) < threshold for key in v1.keys()) line is a bit of a beast. It iterates all the keys in our v1 dictionary, testing abs(v1[k] - v2[k]) for each. Here abs() gives us the absolute or positive difference, so we don't need to compare against negative threshold. Finally, all() tests that this is true for every key we've iterated.

Run this calibrate() function and wiggle the sensor around. You will see the device remain in the calibrating state, with the light flashing, while you wiggle it.

This is because while the device is moving, the difference between consecutive measurements will be greater than the defined threshold.

In the above calibration method we're testing all measurements from the sensor. You could of course only test some of them — e.g. only gyro or acceleration — depending on what you're using.

If you place your sensor onto the table, the calibration test will pass and the function will return values in the same format as for .get_values().

python

>>> calibrate()
{'GyZ': -46, 'GyY': -115, 'GyX': -1937, 'Tmp': 26.8359, 'AcZ': 23960, 'AcY': 44, 'AcX': -872}

The output dictionary of base measurements can be used to adjust subsequent measurements to remove this offset and recalibrate to zero at rest.

Below is an updated get_smoothed_values function which removes the calibrated offset before returning the smoothed data.

python

def get_smoothed_values(n_samples=10, calibration=None):
    """
    Get smoothed values from the sensor by sampling
    the sensor `n_samples` times and returning the mean.

    If passed a `calibration` dictionary, subtract these
    values from the final sensor value before returning.
    """
    result = {}
    for _ in range(n_samples):
        data = accel.get_values()

        for k in data.keys():
            # Add on value / n_samples to produce an average
            # over n_samples, with default of 0 for first loop.
            result[k] = result.get(k, 0) + (data[k] / n_samples)

    if calibration:
        # Remove calibration adjustment.
        for k in calibration.keys():
            result[k] -= calibration[k]

    return result

The following short snippet will allow you to see a table of the gyro and acceleration measurements in (very smoothed) real-time. The numbers are padded to stop them bouncing around as they change, and it uses control-characters to clear the terminal.

python

calibration = calibrate()
while True:
    data = get_smoothed_values(n_samples=100, calibration=calibration)
    print(
        '\t'.join('{0}:{1:>10.1f}'.format(k, data[k])
        for k in sorted(data.keys())),
    end='\r')

Running this you should see something like the following at rest:

python

AcX:     -17.7  AcY:      -3.2  AcZ:      -4.2  GyX:      -1.3  GyY:       1.9  GyZ:       1.8  Tmp:       0.0

If you pick up the sensor, you should see the Z acceleration increase, or decrease as you drop it. The X and Y acceleration should increase/decrease if you tilt the device in any direction. The acceleration measured here is acceleration due to gravity — if you tilt the device so the X axis is pointing straight down, all of g (≈ 9.81 m/s) will be acting through X, and none through Z.

python

AcX:   17679.9  AcY:     233.8  AcZ:  -16332.6  GyX:       3.9  GyY:      32.5  GyZ:       1.0  Tmp:      -0.2

Gyroscopic measurements show rotation around the relevant axis, so will always be zero at rest, but increase with rotational speed in each axis. For example if you rotate the device away from you, you should see a spike in the Y gyroscopic value, which returns to zero as the unit comes to a rest.

python

AcX:   -6540.4  AcY:      22.4  AcZ:   -1930.1  GyX:    -116.7  GyY:     748.9  GyZ:     211.3  Tmp:       0.0

If you pop your finger on the chip, you should also see the temperature raise very slightly.

python

AcX:    -547.8  AcY:      29.8  AcZ:     -36.0  GyX:      -6.9  GyY:       8.9  GyZ:      -3.8  Tmp:       0.3

This covers the basic work of interfacing with an MPU6050 from MicroPython. I'll be adding some projects using this chip shortly.

Dictionary Views & Set Operations — Working with dictionary view objects

2018-09-30T06:00:00+00:00

The keys, values and items from a dictionary can be accessed using the .keys(), .values() and .items() methods. These methods return view objects which provide a view on the source dictionary.

The view objects dict_keys and dict_items support set-like operations (the latter only when all values are hashable) which can be used to combine and filter dictionary elements.

Keys

Dictionary keys are always hashable, so set operations are always available on the dict_keys view object.

All keys (`set` union)

To get all keys from multiple dictionaries, you can use the set union.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}

>>> d1.keys() | d2.keys()
{'key5', 'key3', 'key2', 'key1'}  # this is a set, not a dict

You can use the same approach to combine dict_items and merge dictionaries.

Keys in common (`set` intersection)

The keys common to two dictionaries can be determined using set intersection (&).

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}

>>> d1.keys() & d2.keys()
{'key3'}    # this is a set, not a dictionary

You could use the resulting set to filter your dictionary using a dictionary comprehension.

python

>>> {k:d1[k] for k in keys}
{'key2':'value2', 'key1':'value1'}

Unique keys (`set` difference)

To retrieve keys unique to a given dictionary, you can use set difference (-). Keys from the right hand dict_keys are removed from the left, resulting in a set of the remaining keys.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}

>>> d1.keys() - d2.keys()
{'key1', 'key2'}

Unique keys from both (`set` symmetric difference)

If you want items unique to both dictionaries, the set symmetric difference (^) returns this. The result is items unique to both the left and right hand of the comparison.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}

>>> d1.keys() ^ d2.keys()
{'key5', 'key2', 'key1'}

Items

If both the keys and values of a dictionary are hashable, the dict_items view will support set-like operations.

If the values are not hashable all of these2 operations will all raise a TypeError.

Merge (`set` union)

You can use set union operations to merge two dictionaries.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}
>>> d3 = {'key4':'value4', 'key6':'value6'}

>>> d = dict(d1.items() | d2.items() | d3.items())
>>> d
{'key1':'value1', 'key2':'value2', 'key3':'value3-new', 'key5':'value5', 'key4':'value4', 'key6':'value6'}

Since it is quite common for dictionary values to not be hashable, you will probably want to use one of the other approaches for merging dictionaries instead.

Common entries (`set` intersection)

The items common to two dictionaries can be determined using set intersection (&). Both the key and value must match — items are compared as (key, value) tuples.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key1':'value1', 'key5':'value5'}

>>> d1.items() & d2.items()
{('key1', 'value1')}    # this is a set, not a dict

Unique entries (`set` difference)

To retrieve items unique to a given dictionary, you can use set difference (-). Items from the right hand dict_keys are removed from the left, resulting in a set of the remaining item (key, value) tuples.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}

>>> d1.items() - d2.items()
{('key3', 3), ('key2', 'value2'), ('key1', 'value1')}

Unique entries from both (`set` symmetric difference)

If you want items unique to both dictionaries, the set symmetric difference (^) returns this. The result is item (key, value) tuples unique to both the left and right hand of the comparison.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}

>>> d1.items() ^ d2.items()
{('key2', 'value2'), ('key5', 'value5'), ('key1', 'value1'), ('key3', 3), ('key3', 'value3-new')}

Displaying images on OLED screens — Using 1-bpp images in MicroPython

2018-08-27T16:00:00+00:00

We've previously covered the basics of driving OLED I2C displays from MicroPython, including simple graphics commands and text. Here we look at displaying monochrome 1 bit-per-pixel images and animations using MicroPython on a Wemos D1.

Processing the images and correct choice of image-formats is important to get the most detail, and to not run out of memory.

Requirements
Wemos D1 v2.2+ or good imitations.	amazon
0.96in OLED Screen 128x64 pixels, I2c interface.	amazon
Breadboard Any size will do.	amazon
Wires Loose ends, or jumper leads.

Setting up

The display communicates over I2C, but we need a driver to interface with it. You can import ssd1306 as any other Python module. Connect to your device, and then in the REPL enter:

python

from machine import I2C, Pin
import ssd1306
import framebuf

If the import ssd1306 succeeds, the package is correctly uploaded and you're good to go.

Wire up the OLED display, connecting pins D1 to SCL and D2 to SDA. Provide power from G and 5V. The display below is a 2-colour version, where the top 1/4 of the pixels are yellow, while the rest is blue. They're intended for mobile screens, but it looks kind of neat with Scatman.

python

i2c = I2C(-1, Pin(5), Pin(4))
display = ssd1306.SSD1306_I2C(128, 64, i2c)

If your display is a different size just fiddle the numbers above. You'll need to change some parameters on loops later too.

To test the display is working, let's set all the pixels to on and show it.

python

display.fill(1)
display.show()

The screen should light up completely. If it doesn't, something is wrong.

Image Processing

To display an image on a 1-bit per pixel monochrome display we need to get our image into the same format. The best way to do this is using image manipulation software, such as Photoshop or GIMP. These allow you to down-sample the image to monochrome while maintaining detail by adding dither or other adjustments.

The first step is to crop the image down to the correct dimensions — the display used here is 128x64 pixels. To preserve as much of the image as possible you might find it useful to resize the larger axis to the max (e.g. if the image is wider than high, resize the width to 128 pixels). Then crop the remaining axis.

You can convert images to 1-bit-per-pixel in GIMP through the Image -> Mode -> Indexed... dialog.

If you're image is already in an indexed format this won't be available. So convert back to RGB/Grayscale first, then re-select Image -> Mode -> Indexed.

Select "Use black and white (1-bit) palette" to enable 1bpp mode. The colour dithering settings are best chosen by trial and error depending on the image being converted although turning off dithering entirely is often best for images of solid colour blocks (e.g. logos).

Once the imagine is converted to black & white you can save to file. There are two good options for saving 1bpp images — PBM and PGM. PBM is a 1 bit-per-pixel format, while PGM is grayscale 1 byte per pixel.

Type	Magic number (ASCII)	Magic number (Binary)	Extension	Colors
Portable BitMap	P1	P4	.pbm	0–1 (white & black)
Portable GrayMap	P2	P5	.pgm	0–255 (gray scale)
Portable PixMap	P3	P6	.ppm	0–255 (RGB)

While PBM is clearly better suited, we can pre-process PGM down to an equivalent bit stream. Both approaches are included here, in case your software can only produce one or the other.

Save as either PBM (recommended) or PGM, and select Raw mode, not ASCII.

Example images

Some example images (128x64 pixels) are shown below, in PNG format. Each of the images is available in this zip which contains PBM, PGM and PNG formats.

Portable Bitmap Format

Portable Bitmap Format (PBM) format consists of a regular header, separated by newlines, then the image data. The header starts with a magic number indicating the image format and whether the format is ASCII for binary. In all examples here we're using binary since it's more compact. The second line is a comment, which is usually the program used to create it. Third are the image dimensions. Then, following a final newline, you get the image binary blob.

python

P4
# CREATOR: GIMP PNM Filter Version 1.1
128 64
<data>

The data is stored as a 1-bit-per-pixel stream, with pixel on as 1 pixel off as 0. On a normal display screen an on pixel appears as black — this is different on the OLED, which we need to account for later.

To upload your PBM file to the controller —

bash

ampy --port /dev/tty.wchusbserial141120 put alan.pbm

Loading images

The PBM data stream is already in the correct format for use. We can wrap the data in bytearray, use this to create a FrameBuffer and blit it immediately. However, we need to skip the header region (3x readline) before reading the subsequent data block.

python

with open('scatman.pbm', 'rb') as f:
    f.readline() # Magic number
    f.readline() # Creator comment
    f.readline() # Dimensions
    data = bytearray(f.read())
fbuf = framebuf.FrameBuffer(data, 128, 64, framebuf.MONO_HLSB)

We can't use readlines() since the binary image data may contain ASCII code 13 (newline).

The framebuf.MONO_HLSB format is desribed in the MicroPython docs as —

Monochrome (1-bit) color format This defines a mapping where the bits in a byte are horizontally mapped. Each byte occupies 8 horizontal pixels with bit 0 being the leftmost. Subsequent bytes appear at successive horizontal locations until the rightmost edge is reached. Further bytes are rendered on the next row, one pixel lower.

This matches exactly with the format of our PBM data.

This framebuffer format framebuf.MONO_HLSB used is different to that used by the ssd1306 screen (framebuf.MONO_VLSB). This is handled transparently by the framebuffer when blitting.

Displaying an image

We have the image data in fbuf, which can be blitted directly to our display framebuffer, using .blit. This accepts coordinates at which to blit. Because the OLED screen displays inverse (on = light, off = black) we need to switch .invert(1) on the display.

python

display.invert(1)
display.blit(fbuf, 0, 0)
display.show()

Portable Graymap Format

Portable Graymap Format (PGM) format shares a similar header to PBM, again newline separated. However, there is an additional 4th header line which contains the max value — indicating the number of values between black and white. Black is again zero, max (255 here) is white.

python

P5
# CREATOR: GIMP PNM Filter Version 1.1
128 64
255
<data>

The format uses 1 byte per pixel. This is 8x too many for our purposes, but we can process it down to 1bpp. Since we're saving a mono image each pixel will contain either 0 (fully off) or 255 (fully on).

To upload your PGM file to the controller —

bash

ampy --port /dev/tty.wchusbserial141120 put alan.pgm

Loading images

Since each pixel is a single byte it is easy to iterate, though slow as hell. We opt here to turn on bright pixels, which gives us the correct output without switching the display invert on.

python

with open('alan.pgm', 'rb') as f:
    f.readline() # Magic number
    f.readline() # Creator comment
    f.readline() # Dimensions
    data = bytearray(f.read())

for x in range(128):
    for y in range(32):
        c = data[x + y*128]
        display.pixel(x, y, 1 if c == 255 else 0)

Packing bits

Using 1 byte per pixel wastes 7 bits which is not great, and iterating to draw the pixels is slow. If we pack the bits we can blit as we did with PBM. To do this we simply iterate over the PGM image data in blocks of 8 (8 bits=1 byte).

Each iteration we create our zero'd-byte (an int of 0). As we iterate over the 8 bits, we add 2**(7-n) if that bit should be set to on. The first byte we hit sets the topmost bit, which has a value of 2**(7-0) = 2**7 = 128, the second 2**(7-1) = 2**6 = 64. The table below shows the values for each bit in a byte.

python

|7| 6|5|4|3|2|1|0
|:--|:--|:--|:--|:--|:--|:--|:--|
|2^7|   2^6|2^5|2^4|2^3|2^2|2^1|2^0|
|128|   64|32|16|8|4|2|1|

The result is a single byte with a single bit set in turn for each byte we iterated over.

python

p = []
for i in range(0, len(d), 8):
    byte = 0
    for n, bit in enumerate(d[i:i+8]):
        byte += 2**(7-n) if bit == 255 else 0

    p.append(byte)

We choose to interpret the 255 values as on (the opposite as in PBM where black = on, giving an inverted image). You could of course reverse it.

The variable p now contains a list of int values in the range 0-255 (bytes). We can cast this to a bytearray and then use this create our FrameBuffer object.

python

# Create a framebuffer object
fbuf = framebuf.FrameBuffer(bytearray(p), 128, 64, framebuf.MONO_HLSB)

The framebuf.MONO_HLSB format is desribed in the MicroPython docs as —

Monochrome (1-bit) color format This defines a mapping where the bits in a byte are horizontally mapped. Each byte occupies 8 horizontal pixels with bit 0 being the leftmost. Subsequent bytes appear at successive horizontal locations until the rightmost edge is reached. Further bytes are rendered on the next row, one pixel lower.

This matches exactly with the format of our PGM (and bit-packed) data.

This framebuffer format framebuf.MONO_HLSB used is different to that used by the ssd1306 screen (framebuf.MONO_VLSB). This is handled transparently by the framebuffer when blitting.

Packing script

A command-line packing script is given below (and you can download it here), which can be used to pack a PGM into a 1bpp bitstream. The script accepts a single filename of a PGM file to process, and outputs the resulting packed bit data as <filename>.bin.

python

import os
import sys

fn = sys.argv[1]

with open(fn, 'rb') as f:
    f.readline() # Magic number
    f.readline() # Creator comment
    f.readline() # Dimensions
    f.readline() # Max value, 255
    data = bytearray(f.read())

p = []
for i in range(0, len(data), 8):
    byte = 0
    for n, bit in enumerate(data[i:i+8]):
        byte += 2**(7-n) if bit == 255 else 0

    p.append(byte)

b = bytearray(p)

basename, _ = os.path.splitext(fn)
with open('%s.bin' % basename, 'wb') as f:
    f.write(b)

The resulting file is 1KB in size, and identical to a .pbm format file, minus the header and with colours inverted (this makes display simpler).

bash

python pack.py scatman.1.pgm

ls -l

-rw-r--r--  1 martin  staff  1024 26 Aug 18:11 scatman.bin
-rw-r--r--  1 martin  staff  8245 26 Aug 18:02 scatman.pgm

To upload your BIN file to the controller —

bash

ampy --port /dev/tty.wchusbserial141120 put scatman.bin

Loading images

Since we've stripped off the PGM header, the resulting file can be read directly into a bytearray.

python

with open('scatman.bin', 'rb') as f:
    data = bytearray(f.read())

fbuf = framebuf.FrameBuffer(data, 128, 64, framebuf.MONO_HLSB)

The colours were inverted in our bit packer so we can just blit the framebuffer directly without inverting the display.

python

display.blit(fbuf, 0, 0)
display.show()

Animation

Both the PBM and PGM images are 1KB in memory once loaded, leaving us plenty of space to load multiple images and animate them. The following loads a series of Scatman John PBM images and animates them in a loop.

python

from machine import I2C, Pin
import ssd1306
import time
import framebuf

i2c = I2C(-1, Pin(5), Pin(4))
display = ssd1306.SSD1306_I2C(128, 64, i2c)

images = []
for n in range(1,7):
    with open('scatman.%s.pbm' % n, 'rb') as f:
        f.readline() # Magic number
        f.readline() # Creator comment
        f.readline() # Dimensions
        data = bytearray(f.read())
    fbuf = framebuf.FrameBuffer(data, 128, 64, framebuf.MONO_HLSB)
    images.append(fbuf)

display.invert(1)
while True:
    for i in images:
        display.blit(i, 0, 0)
        display.show()
        time.sleep(0.1)

The resulting animation —

The image distortion is due to frame rate mismatch with the camera and won't be visible in person.

Optimization

There is still plenty of room left for optimization. For static images there are often multiple consecutive blocks of bits of the same colour (think backround regions) or regular patterns (dithering). By setting aside a few bits as repeat markers we could compress these regions down to a single pattern, at the cost of random larger files for very random images and unpacking time.

We could get away with a lot less data for the animation (particularly the example above) by storing only frame deltas (changes), and using key frames. But we'd also need masking, and that takes memory... and yeah. Let's not, for now.

Dictionaries — A rather long guide to Python's key:value hash type

2018-08-26T12:00:00+00:00

Dictionaries are key-value stores, meaning they store, and allow retrieval of data (or values) through a unique key. This is analogous with a real dictionary where you look up definitions (data) using a given key — the word. Unlike a language dictionary however, keys in Python dictionaries are not alphabetically sorted.

From Python 3.6 onwards dictionaries are ordered in that elements are stored and retrieved in the order in which they are added. This usually only has consequences for iterating (see later).

Anything which can be stored in a Python variable can be stored in a dictionary value. That includes mutable types including list and even dict — meaning you can nest dictionaries inside on another. In contrast keys must be hashable and immutable — the object hash must not change once calculated. This means list or dict objects cannot be used for dictionary keys, however a tuple is fine.

A hash is a reproducible, compact, representation of an original value. Reproducible means that hashing the same input will always produce the same output. This is essential for dictionary keys where hashes are used to store and look up values: if the hash changed each time we hashed the key, we'd never find anything!

Creating

Dictionaries can be defined using both literal or constructor syntax. Literal syntax is a bit cleaner, but there are situations where dict() is useful.

python

d = {}        # An empty dictionary, using literal syntax
d = dict()    # An empty dictionary, using object syntax

You can add initial items to a dictionary by passing the key-value pairs at creation time. The following two syntaxes are equivalent, and will produce an identical dictionary.

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}
>>> d
{'key1': 'value1', 'key2': 'value2', 'key3': 3}

>>> d = dict(key1='value1', key2='value2', key3=3)
>>> d
{'key1': 'value1', 'key2': 'value2', 'key3': 3}

However, note that keys in the dict syntax are limited to valid keyword parameter names only — for example, you cannot use anything which would not be a valid variable name (including numbers, number-initial alphanumeric names or punctuation).

python

>>> dict(1='hello')
SyntaxError: invalid syntax

>>> dict(1a='hello')
SyntaxError: invalid syntax

As always in Python, keyword parameters are interpreted as string names, ignoring any variables defined with the same name.

python

>>> a = 12345
>>> {a:'test'}
{12345: 'test'}

>>> dict(a='test')
{'a': 'test'}

For this reason dict() is only really useful where you have very restricted key names. This is often the case, but you can avoid these annoyances completely by sticking with the literal {} syntax.

Adding

You can add items to a dictionary by assigning a value to a key, using the square bracket [] syntax.

python

>>> d = {}
>>> d['this'] = 'that'
>>> d
{'this':'that'}

Assigning to keys which already exist will replace the existing value for that key.

python

>>> d = {}
>>> d['this'] = 'that'
>>> d['this'] = 'the other'
>>> d
{'this':'the other'}

Retrieving

Values for a given key can be retrieved by key, using the square bracket [] syntax.

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}
>>> d['key1']
'value1'

Retrieving an item does not remove it from the dictionary.

python

>>> d
{'key1': 'value1', 'key2': 'value2', 'key3': 3}

The value returned is the same object stored in the dictionary, not a copy. This is important to bear in mind when using mutable objects such as lists as values.

python

>>> d = {'key1': [1,2,3,4]}
>>> l = d['key1']
>>> l
[1,2,3,4]

>>> l.pop()
4

>>> d
d = {'key1': [1,2,3]}

Notice that changes made to the returned list continue to be reflected in the dictionary. The retrieved list and the value in the dictionary are the same object.

Removing

To remove an item from a dictionary you can use del using square bracket syntax with the key to access the element.

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}

>>> del d['key1]
>>> d
{'key2':'value2', 'key3': 3}

You can also remove items from a dictionary by using .pop(<key>). This removes the given key from the dictionary, and returns the value.

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}

>>> d.pop('key1)
'value1'

>>> d
{'key2':'value2', 'key3': 3}

Counting

The number of elements in a dictionary can be found by using len().

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}
>>> len(d)
3

The length of a dictionaries .keys(), .values() and .items() are always equal.

View objects

There are separate view objects for each of keys, values and items — dict_keys, dict_values and dict_items respectively.

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}
>>> d.keys()
dict_keys(['key1', 'key2', 'key3'])

>>> d.values()
dict_values(['value1', 'value2', 3])

dict_items provides a view over tuples of (key, value) pairs.

python

>>> d.items()
dict_items([('key1', 'value1'), ('key2', 'value2'), ('key3', 3)])

These view objects are all iterable. They are also dynamic — changes to the original dictionary continue to be reflected in the view after it is created.

python

>>> k = d.keys()
>>> k
dict_keys(['key1', 'key2', 'key3'])

>>> d['key4'] = 'value4'
>>> k
dict_keys(['key1', 'key2', 'key3', 'key4'])

This is different to Python 2.7, where .keys(), .values() and .items() returned a static list.

Membership

To determine if a given key is present in a dictionary, you can use the in keyword. This will return True if the give key is found, False if it is not.

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}

>>> 'key2' in d
True

>>> 'key5' in d
False

You can also check whether a given value or key-value pair is in a dictionary by using the .values() and .items() views.

python

>>> 'value1' in d.values()
True

>>> 'value5' in d.values()
False

>>> ('key1', 'value1') in d.items()
True

>>> ('key3', 'value5') in d.items()
False

These lookups are less efficient that key-based lookups on dictionaries, and needing to lookup values or items is often an indication that a dict is not a good store for your data.

Lists from dictionaries

To get a list of a dictionary's keys, values or items of a dictionary to lists, we can take the dict_keys, dict_values or dict_items view objects and pass them to list().

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}

>>> list(d.keys())
['key1', 'key2', 'key3']

>>> list(d.values())
['value1', 'value2', 3]

>>> list(d.items())
[('key1', 'value1'), ('key2', 'value2'), ('key3', 3)]

Converting the view objects to lists breaks the link to the original dictionary, so further updates to the dictionary will not be reflected in the list.

Dictionaries from lists

Similarly lists can be used to generate dictionaries. The simplest approach is using a list of 2-tuple where the first element in the tuple is used for the key and the second for the value.

python

>>> l = [('key1', 'value1'), ('key2', 'value2'), ('key3', 3)]
>>> d = dict(l) # Pass the list as to the dict constructor

>>> d
{'key1': 'value1', 'key2': 'value2', 'key3': 3}

You can pass in other iterators, not just lists. The only restriction is that the iterator needs to return 2 items per iteration.

If you have your key and value elements in seperate lists, you can use zip to combine them together into tuples before creating the dictionary.

python

>>> keys = ['key1', 'key2', 'key3']
>>> vals = ['value1', 'value2', 3]

>>> l = zip(keys, vals)
>>> l
<zip object>

>>> dict(l)
{'key1': 'value1', 'key2': 'value2', 'key3': 3}

If key and value lists are not of the same length, the behaviour of zip is to silently drop any extra items from the longer list.

python

>>> keys = ['key1', 'key2', 'oops']
>>> vals = ['value1', 'value2']

>>> dict(zip(keys, vals))
{'key1': 'value1', 'key2': 'value2'}

Iterating

By default iterating over a dictionary iterates over the keys.

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}

>>> for k in d:
...     print(k)
key1
key2
key3

This is functionally equivalent to iterating over the .keys() view.

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}

>>> for k in d.keys():
...     print(k)
key1
key2
key3

The dictionary is unaffected by iterating over it, and you can use the key within your loop to access the value from the dictionary.

python

>>> d = {'key1': 'value1', 'key2': 'value2', 'key3': 3}

>>> for k in d:
...     print(k, d[k])  # Access value by key.
key1 value1
key2 value2
key3 3

If you want access to dictionary values within your loop, you can iterate over items to have them returned in the for loop. The keys vand values are returned as a 2-tuple.

python

>>> d = {'key1':'value1', 'key2':'value2', 'key3':3}

>>> for kv in d.items():
...     print(kv)
('key1', 'value1')
('key2', 'value2')
('key3', 3)

You can unpack the key and value to seperate variables in the loop, making them available without indexing. This is the most common loop structure used with dictionaries.

python

>>> d = {'key1':'value1', 'key2':'value2', 'key3':3}

>>> for k, v in d.items():
...     print(k, v)
key1 value1
key2 value2
key3 3

If you are only interested in the dictionary values you can also iterate over these directly.

python

>>> d = {'key1':'value1', 'key2':'value2', 'key3':3}

>>> for v in d.values():
...     print(v)
value1
value2
3

If you want to count as you iterate you can use enumerate as with any iterator, but you must nest the unpacking.

python

>>> d = {'key1':'value1', 'key2':'value2', 'key3':3}

>>> for n, (k, v) in enumerate(d.items()):
...     print(n, k, v)
0 key1 value1
1 key2 value2
2 key3 3

Dictionary comprehensions

Dictionary comprehensions are shorthand iterations which can be used to construct dictionaries, while filtering or altering keys or values.

Iterating over a list of (key, value) tuples and assigning to keys and values will create a new dictionary.

python

>>> l = [('key1','value1'), ('key2','value2'), ('key3',3)]

>>> {k:v for k,v in l}
{'key1': 'value1', 'key2': 'value2', 'key3': 3}

You can filter elements by using a trailing if clause. If this expression evaluates to False the element will be skipped (if it evaluates True it will be added).

python

>>> l = [('key1','value1'), ('key2','value2'), ('key3',3)]

>>> {k:v for k,v in l if isinstance(v, str)}  # Only add strings.
{'key1': 'value1', 'key2': 'value2'}

Any valid expression can be used for the comparison, as long as it returns thruthy or falsey values.

python

>>> l = [('key1','value1'), ('key2','value2'), ('key3',3)]

>>> {k:v for k,v in l if v != 'value1'}
{'key2': 'value2', 'key3': 3}

Comparisons can be performed against keys, values, or both.

python

>>> l = [('key1','value1'), ('key2','value2'), ('key3',3)]

>>> {k:v for k,v in l if v != 'value1' and k != 'key3'}
{'key2': 'value2'}

Since empty string evaluates as False in Python testing the value alone can be used to strip empty string values from a dictionary.

python

>>> d = {'key1':'value1', 'key2':'value2', 'key3':'', 'another-empty':''}

>>> {k:v for k,v in d.items() if v}
{'key1': 'value1', 'key2': 'value2'}

Separate lists of keys and values can be zipped, and filtered using a dictionary comprehension.

python

>> k = ['key1', 'key2', 'key3']
>> v = ['value1', 'value2', 3]

>>> {k:v for k,v in zip(k,v) if k != 'key1'}
{'key2': 'value2', 'key3': 3}

Expressions can also be used in the k:v construct to alter keys or values that are generated for the dictionary.

python

>>> l = [('key1', 1), ('key2', 2), ('key3', 3)]

>>> {k:v**2 for k,v in l}
{'key1': 1, 'key2': 4, 'key3': 9}

Any expressions are valid, for both keys and values, including calling functions.

python

>>> l = [('key1', 1), ('key2', 2), ('key3', 3)]

>>> def cube(v):
...     return v**3

>>> def reverse(k):
...     return k[::-1]

>>> {reverse(k):cube(v) for k,v in l}
{'1yek': 1, '2yek': 8, '3yek': 27}

You can use a ternary if-else in the k:v to selectively replace keys. In the following example values are replaced if they don't match 'value1'.

python

>>> l = [('key1','value1'), ('key2','value2'), ('key3',3)]

>>> {k:v if v=='value1' else None for k,v in l}
{'key1': 'value1', 'key2': None, 'key3': None}

You can also use ternary syntax to process keys. Any expressions are valid here, in the follow example we replace missing keys with the current iteration number (1-indexed).

python

>>> l = [(None,'value1'), (None,'value2'), ('key3',3)]

>>> {k if k else n:v for n,(k,v) in enumerate(l, 1)}
{1: 'value1', 2: 'value2', 'key3': 3}

If your expressions generate duplicate keys, the later value will take precedence for that key.

python

>>> l = [(None,'value1'), (None,'value2'), ('key3',3)]

>>> {k if k else 0:v for n,(k,v) in enumerate(l)}
{0: 'value2', 'key3': 3} # 0:value1 has been overwritten by 0:value1

You can use nested loops within dictionary comprehensions although you often won't want to since it can get pretty confusing. One useful application of this however is for flattening nested dictionaries. The follow example unnestes 2-deep dictionaries, discarding the outer keys.

python

>>> d = {'a': {'naa':1, 'nab':2, 'nac':3}, 'b': {'nba':4, 'nbb':5, 'nbc':6}}

>>> {k:v for di in d.values() for k,v in di.items()}
{'naa': 1, 'nab': 2, 'nac': 3, 'nba': 4, 'nbb': 5, 'nbc': 6}

The left hand loops it the outer loop, which iterates the d dictionary producing the values in di. The inner loop on the right iterates this dictionary keys and values as k and v, which are used to construct the new dictionary on the far left k:v.

Merging

There are a number of ways to merge dictionaries. The major difference between the approaches is in how (or whether) they handle duplicate keys.

Update

Each dictionary object has an .update() method, which can be used to add a set of keys and values to an existing dictionary, using another dictionary as the source.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key4':'value4', 'key5':'value5'}

>>> d1.update(d2)
>>> d1
{'key1':'value1', 'key2':'value2', 'key3': 3, 'key4':'value4', 'key5':'value5'}

This updates the original dictionary, and does not return a copy.

If there are duplicate keys in the dictionary being updated from, the values from that dictionary will replace those in the dictionary being updated.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}

>>> d1.update(d2)
>>> d1
{'key1':'value1', 'key2':'value2', 'key3':'value3-new', 'key5':'value5'}

If you do not want to replace already existing keys, you can use a dictionary comprehension to pre-filter.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}

>>> d1.update({k:v for k, v in d2.items() if k not in d1})
>>> d1
{'key1':'value1', 'key2':'value2', 'key3': 3, 'key5':'value5'}

Unpacking

Dictionaries can be unpacked to key=value keyword pairs, which is used to pass parameters to functions or constructors. This can be used to combine multiple dictionaries by unpacking them consecutively.

This requires Python 3.6 and above.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key4':'value4', 'key5':'value5'}

>>> d = {**d1, **d2}
>>> d
{'key1': 'value1', 'key2': 'value2', 'key3': 3, 'key4': 'value4', 'key5': 'value5'}

Unpacking using this syntax handles duplicate keys, with the later dictionary taking precedence of the earlier.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}

>>> d = {**d1, **d2}
>>> d
{'key1': 'value1', 'key2': 'value2', 'key3': 'value3-new', 'key5': 'value5'}

You can use this same syntax to merge multiple dictionaries together.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}
>>> d3 = {'key4':'value4', 'key6':'value6'}

>>> d = {**d1, **d2, **d3}
>>> d
{'key1': 'value1', 'key2': 'value2', 'key3': 'value3-new', 'key5': 'value5', 'key4': 'value4', 'key6': 'value6'}

You can also unpack to a dict()

python

>>> dict(**d1, **d3)
{'key1': 'value1', 'key2': 'value2', 'key3': 3, 'key4': 'value4', 'key6': 'value6'}

>>> dict(**d1, **d2)
TypeError: type object got multiple values for keyword argument 'key3'

However, in this case duplicate keys are not supported, and you are limited by the keyword naming restrictions described earlier.

python

>>> dict(**d1, **d2)
TypeError: type object got multiple values for keyword argument 'key3'

>>> dict(**{3:'value3'})
TypeError: keyword arguments must be strings

There is no such restriction for {} unpacking.

python

>>> {**{3:'value3'}}
{3:'value3'}

Addition (Python 2.7 only)

In Python 2.7 dict.items() returns a list of (key, value) tuples. Lists can be concatenated using the + operator, and the resulting list can be converted back to a new dictionary by passing to the dict constructor.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}

>>> l = dict(d1.items() + d2.items())
>>> l
[('key3', 3), ('key2', 'value2'), ('key1', 'value1'), ('key3', 'value3-new'), ('key5', 'value5')]

>>> dict(l)
{'key3': 'value3-new', 'key2': 'value2', 'key1': 'value1', 'key5': 'value5'}

You can add together multiple dictionaries using this method. The later dictionary keys take precedence over the former.

Union (set merge)

If both the keys and values of a dictionary are hashable, the dict_items view supports set-like operations.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5':'value5'}
>>> d3 = {'key4':'value4', 'key6':'value6'}

>>> dict(d1.items() | d2.items() | d3.items())
{'key4': 'value4', 'key5': 'value5', 'key2': 'value2', 'key6': 'value6', 'key3': 3, 'key1': 'value1'}

The merging occurs right-to left.

If the values are not hashable this will raise a TypeError.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}
>>> d2 = {'key3':'value3-new', 'key5': []}  # list is unhashable

>>> d1.items() | d2.items()
TypeError: unhashable type:'list'

All standard set operations are possible on dict_keys and dict_items.

Copying

To make a copy of an existing dictionary you can use .copy(). This results in an identical dictionary which is a distinct object.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}

>>> d2 = d1.copy()
>>> d2
{'key1':'value1', 'key2':'value2', 'key3':3}

>>> id(d1) == id(d2)
False

You can also make a copy of a dictionary by passing an existing dictionary to the dict constructor. This is functionally equivalent to .copy().

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':3}

>>> d2 = dict(d1)
>>> d2
{'key1':'value1', 'key2':'value2', 'key3':3}

>>> id(d1) == id(d2)
False

In both cases these are shallow copies meaning nested objects within the dictionary are not also copied. Changes to this nested objects will also be reflected in the original dictionary.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':{'nested':'dictionary'}}

>>> d2 = d1.copy()
>>> d2
{'key1': 'value1', 'key2': 'value2', 'key3': {'nested': 'dictionary'}}

>>> id(d1) == id(d2)
False

>>> id(d1['key3']) == id(d2['key3'])
True

>>> d2['key3']['nested'] = 'I changed in d1'
>>> d1
{'key1': 'value1', 'key2': 'value2', 'key3': {'nested': 'I changed in d1'}}

If you want nested objects to also be copied, you need to create a deepcopy of your dictionary.

python

>>> d1 = {'key1':'value1', 'key2':'value2', 'key3':{'nested':'dictionary'}}

>>> from copy import deepcopy
>>> d2 = deepcopy(d1)
>>> d2
{'key1': 'value1', 'key2': 'value2', 'key3': {'nested': 'dictionary'}}

>>> id(d1) == id(d2)
False

>>> id(d1['key3']) == id(d2['key3'])
False

>>> d2['key3']['nested'] = ['I did not change in d1']
>>> d1
{'key1': 'value1', 'key2': 'value2', 'key3': {'nested': 'dictionary'}}

Since a deepcopy copies all nested objects it is slower and uses more memory. Only use it when it's actually neccessary.

Driving I2C OLED displays with MicroPython — I2C monochrome displays with SSD1306

2018-08-25T08:00:00+00:00

These mini monochrome OLED screens make great displays for projects — perfect for data readout, simple UIs or monochrome games.

Requirements
Wemos D1 v2.2+ or good imitations.	amazon
0.91in OLED Screen 128x32 pixels, I2c interface.	amazon
Breadboard Any size will do.	amazon
Wires Loose ends, or jumper leads.

Setting up

MicroPython provides some built-in support for these displays, and a simple framebuffer which can be used as a drawing surface. However, we still need a driver to interface the buffer to the display.

There is a Python ssd1306 module for OLED displays available in the MicroPython repository. Click Raw format and save the file with a .py extension.

You can then use the ampy tool (or the WebREPL) to upload the file to your device's filesystem:

bash

ampy --port /dev/tty.wchusbserial141120 put ssd1306.py

With the ssd1306.py file on your Wemos D1, you can import it as any other Python module. Connect to your device, and then in the REPL enter:

python

from machine import I2C, Pin
import ssd1306

Both I2C and SPI-controlled displays are available — these instructions will only work for I2C displays.

If the import ssd1306 succeeds, the package is correctly uploaded and you're good to go. Wire up the OLED display, connecting pins D1 to SCL and D2 to SDA. Provide power from G and 5V.

Using the interface

The following instructions are based on a 0.91in monochrome 128x32 OLED, but other displays using the same chipset can be used with this library.

To use the display you first need to create an I2C interface. In MicroPython I2C is via a software implementation so you can put it on any GPIO pins you like.

python

i2c = I2C(-1, Pin(5), Pin(4))
display = ssd1306.SSD1306_I2C(128, 32, i2c)

The SSD1306 module makes use of the MicroPython framebuf frame buffer, an efficient in-memory buffer for working with a simple graphics view. The methods for drawing text and primitives are from this framebuffer implementation. For a complete overview of what's available, check the MicroPython documentation.

The setup of the framebuffer format (monochrome, bit ordering, etc.) is also handled by the SSD1306 library. Check the framebuf documentation for more info on available options for other displays.

To test your I2C connection to the display, fill the display in solid colour.

python

display.fill(1)  # Fill the entire display with 1="on"
display.show()

You need to call display.show() to actually send the current framebuf to the device.

Drawing primitives

The .fill() method can be used to fill the entire display with a specified colour. Note that this changes all pixels to the given colour, and does not perform a flood fill of matching regions.

python

# Fill the entire display with colour 0
display.fill(0)

For filling specific regions of the display see .fill_rect() below.

Setting individual pixels can be accomplished using .pixel(). This is only a good idea when you area setting relatively few pixels as it is much slower than using other methods.

python

# Set the pixel at 3, 4 (x, y) to 1
# .pixel(x, y, c)
display.pixel(3, 4, 1)  # 3rd param is the colour
display.show()

If you don't provide a color via parameter c, this method returns the colour of the pixel at the specified coordinates.

python

# Return the value at 3, 4 (x, y)
# .pixel(x, y)
c = display.pixel(3, 4)

This is not a particularly nice API. In other cases (e.g. text) omitting the optional c will use 1 as a default.

Horizontal and vertical lines can be drawn with .hline() and .vline() respectively, providing a starting x,y location and line length and colour.

python

# Draw a horizontal line, starting from 2, 3 (x, y), 4 pixels wide
# .hline(x, y, w, c)
display.hline(2, 3, 25, 1)
display.show()

python

# Draw a vertical line, starting from 5, 0 (x, y), 6 pixels high
# .vline(x, y, h, c)
display.vline(5, 0, 15, 1)
display.show()

For diagonal lines, the .line() method can be used to draw lines between between two sets of points x1,y1 and x2,y2 specified in order. The parameter c controls the colour of the line drawn.

python

# Draw a short diagonal line down to the right.
# .line(x1, y1, x2, y2, c)
display.line(0, 0, 50, 25, 1)
display.show()

There is no antialiasing, so diagonal lines will probably look pretty jaggy.

The .draw_rect() method allows you to draw a unfilled rectangle, starting at x,y and with a specified width w and height h. The specified colour c is used to draw the boundary of the rectangle, but it is not filled.

python

# Draw an unfilled rectangle of 8, 5 pixels, starting at 1,1 in colour 1.
# .rect(x, y, w, h, c)
display.rect(5, 5, 100, 20, 1)
display.show()

You can also draw filled rectangles, using .fill_rect(). The parameters are the same as for .rect() but all pixels within the boundary will be set.

python

# Draw a filled rectangle of 10x5 pixels, starting at 3,3 in colour 1
# .fill_rect(x, y, w, h, c)
display.fill_rect(9, 9, 25, 25, 1)
display.show()

Writing text

The framebuffer class provides support for writing text using a simple 8x8 bitmap font. The pixel x,y positions are relative to the top-left of the 8x8 character, so positioning at 0,0 will give the absolute top left of the screen.

python

# Print "Hello world!" 1 pixel from top left, in colour 1 (on)
# .text(text, x, y, c)
display.text("Hello world!", 1, 1, 1)
display.show()

The .text() method takes an optional 4th parameter color, which gives the colour to draw text in. On a mono screen this can be either 0 (off) or 1 (on).

python

# Print "Hello world!" at 2,2 in colour 0 (off)
display.text("Hello world!", 2, 2, 0)
display.show()

python

# Print "Hello world!" at 3,3 in colour 1 (on)
display.text("Hello world!", 3, 3, 1)
display.show()

Scrolling

You can shift the contents of the entire framebuffer around using .scroll, which takes x and y parameters to specify the scroll (positive/negative) in each dimension.

python

# .scroll(x, y)
display.scroll(-10, 0)

There is no support for scrolling a portion of the framebuffer. You can however keep a separate framebuffer which you scroll and then blit to your main display — see the next section.

Pixel graphics

For very simple graphics that do not update often you can get away with writing bitmap graphics to the framebuffer pixel by pixel. For example, in the following code block we draw an icon from a list-of-lists of binary colour data, iterating over with simple loops:

python

ICON = [
    [ 0, 0, 0, 0, 0, 0, 0, 0, 0],
    [ 0, 1, 1, 0, 0, 0, 1, 1, 0],
    [ 1, 1, 1, 1, 0, 1, 1, 1, 1],
    [ 1, 1, 1, 1, 1, 1, 1, 1, 1],
    [ 1, 1, 1, 1, 1, 1, 1, 1, 1],
    [ 0, 1, 1, 1, 1, 1, 1, 1, 0],
    [ 0, 0, 1, 1, 1, 1, 1, 0, 0],
    [ 0, 0, 0, 1, 1, 1, 0, 0, 0],
    [ 0, 0, 0, 0, 1, 0, 0, 0, 0],
]

display.fill(0) # Clear the display
for y, row in enumerate(ICON):
    for x, c in enumerate(row):
        display.pixel(x, y, c)

display.show()

Using urandom we can scatter images over the display, like in the following example:

python

import urandom

def random_heart():
    xofs = urandom.getrandbits(8)
    yofs = urandom.getrandbits(5)
    for y, row in enumerate(ICON):
        for x, c in enumerate(row):
            display.pixel(x + xofs, y + yofs, c)

for n in range(100):
    random_heart()

display.show()

If you can add an if c around the display.pixel call to only output pixels which are on — effectively masking, and avoiding the black square.

For bigger graphics or where you need faster updates, you can instead blit your bitmap images from one framebuffer to another. Pass the framebuf to blit, and the coordinates x and y to blit at.

python

# Blit a framebuffer at the pixel position 1, 1.
# display.blit(fbuf, x, y)
display.blit(fbuf, 1, 1)

The blit method takes an optional 4th parameter key which is a colour to considered 'transparent' when blitting. Pixels with this value won't be copied over onto the target display.

python

# Blit except pixels with value 0 (i.e. only pixels with value 1)
# .blit(fbuf, x, y, key)
display.blit(fbuf, 1, 1, key=0)

See this follow up for details on how to display images on OLED screens, including animated graphics.

Display control

The drawing methods so far are inherited from the the MicroPython framebuf object, which is written to the display on .update(). But the ssd1306 object itself also provides methods for direct control of the display component.

The display can be turned on and off using display.poweron() and display.poweroff() respectively.

You can set the contrast for the display using .contrast() passing in parameter c which is a value between 0 and 255. This controls the contrast between the foreground, active colour 1 and the background 0.

python

# Set the display contrast to half (127/255 = 0.5)
# .contrast(c)
display.contrast(50)

To invert the display, switching foreground and background colours, call .invert(1).

python

display.invert(1)

Calling .invert(0) will return the display to how it was originally. You can use this for display-flashing visual effects:

python

import time
while True:
    display.invert(0)
    time.sleep(0.01)
    display.invert(1)
    time.sleep(0.01)

Heart rate (HR) sensors — Photoplethysmography, because that's a real word

2018-02-27T00:00:00+00:00

Pulse sensors are a common feature of fitness monitors, used to track your activity and cardiac fitness over time. These external monitors use the reflection and absorption of bright green/infra-red light to detect the pulse wave travelling down the artery — a technique called photoplethysmography (PPG). This same techique is used in hospital finger-clip monitors. Wrist-worn devices like the Fitbit typically use green-light sensors, while the IPhone monitor uses a combination of green and infra-red light. The use of infra-red light for pulse monitoring has a longer history, and is the type of choice used in hospital monitors, because (in combination with a red LED) allows both heart rate and oxygen saturation sensing.

In healthy individuals the oxygen saturation level shouldn't fall, so it's not particularly useful in a fitness trcker.

How it works

Haemoglobin, the oxygen-carrying component of red blood cells, is a strong reflector of red light, and a strong absorber of green light. Together these characteristics give oxygenated blood its red colour.

The image below (adapted from Wikimedia) shows the relative absorption by blood of different wavelengths of light. Green, red and infra-red regions are highlighted.

The differences in absorption between oxygenated (red) and deoxygenated (blue) blood can be used with paired red-IR LEDs to determine blood oxygenation percentage. See the MAX30100 sensor for an example.

During a pulse there is a wave of increase blood pressure through the arteries, slightly stretching the elastic arterial walls and bringing a pulse of highly oxygenated blood. This change in arterial size and increase concentration of blood is what is exploited to detect the pulse with PPG.

Green light is absorbed by red blood cells, but scattered by the tissues. Between pulses, scattered light will be reflected back out towards the incident light and can be detected. However, during a pulse, the small increase in blood volume leads to an increased absorption of the green light. This results in a reduction of reflected signal. We cab detect this increased absorbance by the reduction in reflection of green light.

Red/infra-red light is reflected by red blood cells. Between pulses most light is transmitted and scattered into the tissues. During a pulse the small increase in blood volume leads to an increased reflection of light by red blood cells, and a reduction in transmission. We can detect this increased scattering of light either by the reduction in IR transmission, or alternatively an increase in reflection.

Below we look at the main types and how to interface with them from different microcontrollers. You can also use one of these sensors to build a working heart monitor.

Analogue Sensors (KY-039/Pulsesensor.com)

Sensor types

IR-phototransistor sensors (KY-039) like the Keyes KY-039 use a bright infrared (IR) LED and a photo transistor to detect the reduced IR transmission or increased IR reflection during the pulse. These sensors are more suited to transmission use because of their construction.

Green-light phototransistors like the PulseSensor.com function in reverse usign reflected, rather than transmitted light to detect the pulse. This is done using a rather pleasant green LED, which when wired up looks a bit like a tripod from the War of the Worlds. By using green light these sensors detect the reduction in reflection due to blood absorbing the green light. The circuitry for these sensors is a little more sophisticated than a rawlight sensor, with automatic amplification and noise cancellation features. This makes the subsequent HR calculation a little simpler. The sensors are similar to those found in wearable heart rate/fitness devices.

See this tutorial for an example of building a heart rate monitor using a Pulsesensor.com sensor.

Reading analogue sensors

The principal for reading both types of sensor is the same. With the sensor + connected to Vref (3.3V or 5V; depending on controller), and GND to GND, the measured value is readable via the S pin. The analogue output from this pin varies between GND (light completely blocked) and Vref (light completely passing).

The variation caused by the pulse is a tiny fraction of that caused by putting your finger in front of the sensor, so calculating the pulse value requires a bit of code (see later).

Raspberry Pi

Wire the + on the sensor to +3.3v on your Pi. While the sensor can handle 5V itself, the input voltage sets the max output from the S pin. More than 3.3V will damage your Pi's GPIO.

As the readings are analogue you will need a Analogue to Digital Converter (ADC) such as the MCP3008 between the sensor and your Pi. The Pi then communicates with the chip via SPI.

Add the MCP3008 to a breadboard, with the notch facing up towards your breakout board. Using hardware SPI we can wire the MCP3008 up as follows:

MCP3008	Raspberry Pi
`VDD` (16)	`3.3V`
`VREF` (15)	`3.3V`
`AGND` (14)	`GND`
`DGND` (13)	`GND`
`CLK` (12)	`SCLK` (11)
`DOUT` (11)	`MISO` (9)
`DIN` (10)	`MOSI` (10)
`CS/SHDN` (9)	`CE0` (8)

The analogue and digital ground pins are wired together here. This is fine since we aren't looking for highly accurate readings.

Wire the 3.3V and GND pins up first, leaving the SCLK, MISO, MOSI and CEO pins to wire to your Pi. They connect to pins 11, 9, 10 and 8 respectively, with the first 3 in order up one side of the GPIO, and the last on the other side, level with the first pin you connected.

Once connected you can use the gpiozero.MCP3008() interface to read values out directly.

python

import gpiozero
hr = gpiozero.MCP3008(channel=0)  # Set the channel if you're using a different one

>>> hr.value
    0.7663

To check that the sensor is responding correctly, try writing the values out in a loop (using end="\r" keeps our output on a single line).

python

while True:
    print("%4f" % hr.value, end="\r")

If you put your finger between the emitter and sensor while running this you should see the number increase, indicating increased resistance as the light has to travel through your finger. Somewhere buried in the noisy variation you see is your heart beating.

MicroPython (ESP8266 inc. Wemos D1)

If you're running on a ESP8266 device you have access to an ADC pin, labelled as A0 on the Wemos D1. Once your sensor is wired to this pin you can use following MicroPython code to read the value from the sensor:

python

import machine
adc = machine.ADC(0)

>>> adc.read()
550

The values output by this ADC are integers in the range 0-1023. A reading of 550 is equivalent to 0.54 on a 0-1 scale.

Below is a plot of data from the Pulsesensor.com sensor, as read using a Wemos D1. Notice that the Pulsesensor.com sensor auto-calibrates the reading range.

BBC micro:bit

The micro:bit can read analog values on pins 0, 1, 2, 3, 4 and 10.

python

from microbit import *
>>> pin0.read_analog()
550

The values output by this ADC are integers in the range 0-1023. A reading of 550 is equivalent to 0.54 on a 0-1 scale.

Digital sensors (MAX30100/RCWL-0530)

Digital sensors based on MAX30100 chips offer both heart rate sending and oximetry (oxygen saturation measurements). These include 2 LEDs (IR and visible red light) to detect pulse and oxygen saturation levels respectively.

Oximeters pair two light emitters and receivers working at slightly different wavelengths. Using the different light absorbance profiles of oxygenated and deoxygenated blood it is possible to determine the level of O₂ in the blood in addition to the pulse.

Reading digital sensors

The MAX30100 chip datasheet provides an I2C interface to read the resulting data, which can be interfaced directly to any I2C supporting micro controller. It is available at I2C channel 0x57 (87).

There was no simple interface available for connecting with a MAX30100 based sensor from a Raspberry Pi (the Intel IoT UMP project looks promising, but doesn't build on Pi as of September 2017). As a stop-gap I re-implemented a C library in Python. The source code is available. This currently works with Raspberry Pi only, although adapting to MicroPython should be as simple as writing an adaptor for the I2C API.

You can create an interface to your device/chip by creating an instance of the MAX30100 class.

python

import max30100
mx30 = max30100.MAX30100()

To read the sensor use .read_sensor(). This stores the current measurement, making this value available under two properties. Historic values are in a queue.

python

mx30.read_sensor()

# The latest value is now available by .ir
mx30.ir

You can switch on O₂ sensing as follows.

python

ms30.enable_spo2()

Subsequent calls to .read_sensor() will now populate the .red attribute.

python

# The latest value is now available by .ir and .red
mx30.ir, mx30.red

For more information on using the max30100 package, including the O₂ saturation sensor, see the documentation on Github.

Calculating a heart rate

Regardless of what sensor you use, you will end up with the same output — a stream of values, indicating transmission or reflection of a particular wavelength of light.

There are a number of ways to calculate the heart rate from this data, but peak-counting is a simple and effective method. For a working example of how to do this see this Wemos D1 heart rate monitor.

Partial Least Squares Discriminant Analysis (PLS-DA) with Python

2015-10-18T09:00:00+00:00

Partial least squares discriminant analysis (PLS-DA) is an adaptation of PLS regression methods to the problem of supervised clustering. It has seen extensive use in the analysis of multivariate datasets, such as that derived from NMR-based metabolomics.

In this method the groups within the samples are already known (e.g. experimental groups) and the goal therefore is to determine two things —

Are the groups actually different?
What are the features that best describe the differences between the groups?

In an experimental context this means determining whether control and test samples are different, and identifying the (known, quantified) experimental variables that contribute to that difference.

PLS-DA is based on PLS regression (PLS-R) with the Y variable generated from experimental group membership, mapped into a linear space. In a 2-group experiment this can be as simple as 0 and 1.

Setting up

The implementation of PLS we will be using is provided by the scikit-learn library. We will also be making use of matplotlib for plotting our outputs and pandas for some basic data handling.

pip install scikit-learn matplotlib pandas

The sample data for this example is available for download

Download and unzip the file into your data folder.

For this demo we will start with 1D ¹H NMR data as it makes explanation and visualization of the PLS models easy to understand. However, later we will also generate PLS-DA models for other data types, to demonstrate how you can easily apply these same methods to any type of multivariate data set.

Loading the data

Before starting, let's take a look at the data we are working with. Create a new Jupyter notebook using the Python 3 kernel, and in the first cell enter and run the following. This will import all the neccessary libraries, as well as using the %matplotlib magic to display output figures in the notebook.

python

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sklearn
import os
plt.style.use('ggplot')

We will start by loading the data. The source data is provided in CSV format with experimental samples along the horizontal axis and spectral variables (ppm) along the vertical axis. In addition to the a sample number, there is also a sample group (class) from the experiment).

There are many ways to load this data, but using pandas allows us to keep the elements of the data together nicely.

python

df = pd.read_csv('data.csv'), index_col=0, header=[0,1])
df

Visualising the dataset

Let's plot the raw data to begin with to get an idea of what we're working with. We can use the build in pandas plot functions to do this quickly.

python

df.plot(kind='line',legend=False, figsize=(12,4))

If you look closely you'll see most of the samples look very alike, but there is one (in red) that looks very unusual. This will become important later.

You might be interested to see experimental groups plotted the same color. The information on experimental groups is stored in the Pandas column MultiIndex 'Group', and can be retrieved using:

python

df.columns.get_level_values('Group')

Index([u'H', u'N', u'N', u'N', u'N', u'N', u'N', u'N', u'N', u'N', u'H', u'H',
        u'H', u'H', u'H', u'H', u'H'],
        dtype='object', name=u'Group')

As this is basically a list of values ("N" or "H", one for each sample) we can use this, together with a dictionary colormap, to plot samples from each group in a single colour.

python

colormap = {
    'N': '#ff0000',  # Red
    'H': '#0000ff',  # Blue
}

colorlist = [colormap[c] for c in df.columns.get_level_values('Group')]

df.plot(kind='line', legend=False, figsize=(12,4), color=colorlist)

Now we can see that the dodgy sample is from the "H" (blue) group.

Building the model

Let's now perform the PLS-DA. As already described, to do this we need to create a pseudolinear Y value against which to correlate the samples. Since we have only two sample groups we can do this very easily.

python

y = [g == 'N' for g in df.columns.get_level_values('Group')]
y


[False,
    True,
    True,
    True,
    True,
    True,
    True,
    True,
    True,
    True,
    False,
    False,
    False,
    False,
    False,
    False,
    False]

Next we convert the boolean values into 0 and 1 by setting the dtype on the array.

python

y = np.array(y, dtype=int)
y


array([0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0])

If you have more than 2 groups, but want to perform a similar 2-group analysis, you can filter the DataFrame using Pandas indexing.

Next we apply fit a PLS-DA model to our data. We must specify the number of components, or latent variables (LVs), to use for our data.

python

from sklearn.cross_decomposition import PLSRegression
plsr = PLSRegression(n_components=2, scale=False) # (1)
plsr.fit(df.values.T, y) # (2)

We select 2 components, with autoscaling off.
The algorithm expects data to be in the transpose, with samples in the horizontal axis and features in the vertical.

Scores and weights

The key elements of a PLS-DA are the scores and weights of the model. The scores describe the position of each sample in each determined latent variable (LV). Notice how there are two columns, one for each LV, and 17 rows, one for each sample. You can see the scores as follows:

python

plsr.x_scores_

array([
    [-3.89611177,  0.63670615],
    [ 0.45130405,  0.2670518 ],
    [ 0.57231162,  0.21282613],
    [ 0.49461256,  0.14515445],
    [ 0.56241956,  0.09096893],
    [ 0.6650397 ,  0.27074594],
    [ 0.49613275,  0.35219217],
    [ 0.10264215,  0.51124086],
    [ 0.58655054,  0.27135712],
    [ 0.73518884,  0.11557932],
    [ 0.58786011,  0.08883542],
    [-0.64436773, -0.88135558],
    [-0.5939848 , -0.83145755],
    [-0.5002382 , -0.81607976],
    [ 0.13732602, -0.17700316],
    [ 0.17014101, -0.15633072],
    [ 0.07317358, -0.10043153]
])

In contrast, the weights describe the contribution of each variable to each LV. You can view the shape of the weights as follows. Again there are 2 columns, one for each LV and 383 rows, one for each variable.

python

plsr.x_weights_.shape

You can remove the .shape to see the full list of data, but it's rather long.

This isn't very enlightening in itself so lets plot these values. Again, we'll generate a DataFrame to simplify the plotting. As samples are in the rows in the scores array, we assign an index from df.columns.

python

scores = pd.DataFrame(plsr.x_scores_)
scores.index=df.columns

ax = scores.plot(x=0, y=1, kind='scatter', s=50, alpha=0.7,
                    c=colorlist, figsize=(6,6))
ax.set_xlabel('Scores on LV 1')
ax.set_ylabel('Scores on LV 2')

So we have good separation between our sample groups, however it is in LV2. While practically this has little effect, the fact that the major variation in our data is not between our sample groups is a little alarming. Remember, PLS-DA is a supervised approach, whereby the algorithm is specifically tuned to find the difference beteen the groups. So, what is happening?

If you think back to when we first plotted the data, there was an ugly looking sample in the dataset — a blue spectra that was very different to all the others, with artefacts and bumps. The variation in this sample is so large that it is swamping the difference between our samples.

So, which sample is it. Let's look at the scores DataFrame.

python

scores

0         1
Sample Group
101    H     -3.896112  0.636706
103    N      0.451304  0.267052
105    N      0.572312  0.212826
107    N      0.494613  0.145154
109    N      0.562420  0.090969
111    N      0.665040  0.270746
113    N      0.496133  0.352192
115    N      0.102642  0.511241
117    N      0.586551  0.271357
119    N      0.735189  0.115579
85     H      0.587860  0.088835
89     H     -0.644368 -0.881356
91     H     -0.593985 -0.831458
93     H     -0.500238 -0.816080
95     H      0.137326 -0.177003
97     H      0.170141 -0.156331
99     H      0.073174 -0.100432

We're looking for a sample with a score in LV1 (column 0) of ~ -4.0. The only sample in the scores with that sort of value is Sample=101, in group "H". We could also plot sample numbers on the figure as follows:

python

scores = pd.DataFrame(plsr.x_scores_)
scores.index=df.columns

ax = scores.plot(x=0, y=1, kind='scatter', s=50, alpha=0.7,
                    c=colorlist, figsize=(6,6))
ax.set_xlabel('Scores on LV 1')
ax.set_ylabel('Scores on LV 2')

for n, (x, y) in enumerate(scores.values):
    label = scores.index.values[n][0]
    ax.text(x,y,label)

Iterating over scores.values would return a tuple of values, one for each column. This means we can do for x,y in scores.values: to get the x,y values. However, we also need the index (row) in order to get the label for the sample. To get an index, we wrap the scores.values in enumerate. Each loop generates a tuple, containing the index and the tuple of x,y which is then unpacked by (n, (x, y).

Now we know which sample is causing the problems, let's remove it.

python

f_df = df.iloc[:, df.columns.get_level_values('Sample') != '101']
f_df

We can do repeat the analysis so far, with the excluded value. Note that we also need to update the colormap.

python

f_colorlist = [colormap[c] for c in f_df.columns.get_level_values('Group')]
f_y = np.array([g == 'N' for g in f_df.columns.get_level_values('Group')], dtype=int)

from sklearn.cross_decomposition import PLSRegression
f_plsr = PLSRegression(n_components=2, scale=False)
f_plsr.fit(f_df.values.T, f_y)

Let's plot the PLS-DA model built using filtered values.

python

f_scores = pd.DataFrame(f_plsr.x_scores_)
f_scores.index=f_df.columns

ax = f_scores.plot(x=0, y=1, kind='scatter', s=50, alpha=0.7,
                    c=f_colorlist, figsize=(6,6))
ax.set_xlabel('Scores on LV 1')
ax.set_ylabel('Scores on LV 2')

for n, (x, y) in enumerate(f_scores.values):
    label = f_scores.index.values[n][0]
    ax.text(x,y,label)

This looks a lot better, the majority of our variation is now in the first latent variable. You'll notice that one of our samples is still mis-classified, but that's just science. As there is nothing objectively wrong with this sample, we just have to accept that our experiment wasn't perfect.

However, if we did want to filter that sample from the DataFrame we could update our filter code as follows:

python

samples_to_filter = ['85','101']
filter_ = [s not in samples_to_filter for s in df.columns.get_level_values('Sample')]
ff_df = df.iloc[:, filter_ ]

1D 1H NMR data processing

2015-10-17T09:00:00+00:00

1D ¹H NMR is a commonly applied to metabolomic studies, being well suited to untargeted analysis of complex biofluids. It has been successfully applied to the classification and diagnosis of a number of diseases including [ref].

There are a number of important steps that must be applied prior 1D ¹H NMR data to statistical analysis, all gearded towards ensuring that the variation observed in the data is representative of real biological variation and not a artifact of sample handling, metabolite extraction or NMR acquisition. These steps are described briefly below and approached in turn in the following method. It is important to familiarise yourself with both the purpose of these steps, and any associated caveats or limits, so that you can correctly apply them to your own data.

Setting up

To process raw NMR data we will be making use of of the nmrglue library. In addition we will make use of features from the numpy, scipy and matplotlib libraries, which you should already have installed. To make sure you can run the following command on the terminal:

python

pip install numpy scipy nmrglue matplotlib

The demo data for this example is available for download.

Download and unzip the file into your data folder.

Loading Bruker FID spectra

Create a new Jupyter notebook using the Python 3 kernel, and in the first cell enter and run the following. This will import all the neccessary libraries, as well as using the %matplotlib magic to display output figures in the notebook.

python

%matplotlib inline
import matplotlib.pyplot as plt
import nmrglue as ng
import numpy as np
import scipy as sp
import os
plt.style.use('ggplot')

Bruker-format data is processed using a digital filter before saving as FID, this filter must be removed for subsequent analysis to work. To open a given spectra, pass the parent folder of the fid file.

python

dic, data = ng.fileio.bruker.read('87')
data = ng.bruker.remove_digital_filter(dic, data)

We can plot the spectra using matplotlib.

python

fig = plt.figure(figsize=(12,4))
ax = fig.add_subplot(1,1,1)
ax.plot(np.arange(0, data.shape[0]), data) # <1>

We don't have the x-axis ppm scale yet, so we generate a linear scale of the same size as the data. Here .shape[0] is the length of the fid along the first (and only) axis of the 1D spectra.

Fourier transform (FT)

A fourier transform is a signal transformation that decomposes a signal into it's constituent frequencies. In the case of NMR data, this decomposition is to a series of peaks, that represent the resonance of chemical subgroups. These peaks contain both our identification (frequency) and quantification (amplitude) information.

The nmrglue package provides a fast fourier transform (FFT) for this purpose

python

data_fft = ng.proc_base.fft(data)

fig = plt.figure(figsize=(12,4))
ax = fig.add_subplot(1,1,1)
ax.plot(np.arange(0, data_fft.shape[0]), data_fft)

You'll notice that the spectra is out of phase, i.e. the peaks are not symmetrical and well defined. We will fix this shortly, but first lets load a complete set of spectra so we can view them all at once.

Batch loading + FFT for multiple spectra

python

root_folder = '.'
zero_fill_size = 32768
data = [] # <1>
for folder in os.listdir(root_folder):
    dic, fid = ng.fileio.bruker.read(os.path.join(root_folder,folder))
    fid = ng.bruker.remove_digital_filter(dic, fid)
    fid = ng.proc_base.zf_size(fid, zero_fill_size) # <2>
    fid = ng.proc_base.rev(fid) # <3>
    fid = ng.proc_base.fft(fid)
    data.append(fid)
data = np.array(data)

We build as a list, then transform to an array for speed.
Zero-filling to a power of 2 speeds up FFT.
Reverse the data to match common representation (for convenience only).

The data is output in a 2D array, with samples along the first axis.

python

data.shape

(17, 32768)

We want to be able to plot multiple spectra on the same plot, so we'll write a simple function to do this.

python

def plotspectra(ppms, data, start=None, stop=None):

    if start: # <1>
        ixs = list(ppms).index(start)
        ppms = ppms[ixs:]
        data = data[:,ixs:]
    if stop:
        ixs = list(ppms).index(stop)
        ppms = ppms[:ixs]
        data = data[:,:ixs]


    fig = plt.figure(figsize=(12, 4))
    ax = fig.add_subplot(1,1,1)
    for n in range(data.shape[0]):
        ax.plot(ppms, data[n,:])

    ax.set_xlabel('ppm')
    ax.invert_xaxis()

    return fig

This block allows us to select regions of the spectra to view by filtering on the supplied ppms.

We can use this function to look at all spectra following fourier transform:

python

ppms = np.arange(0, data.shape[1])
fig = plotspectra(ppms, data)

We can take a close up view of the TMSP peak, which is currently on the left hand side of the spectra in the region 28000-30000.

python

ppms = np.arange(0, data.shape[1])
fig = plotspectra(ppms, data, start=28000, stop=30000)

Clearly the spectra are all out of phase and poorly aligned. We will fix the phasing problem first, but lets start off by getting the correct ppm values for the spectra.

Calculating ppm values

The method for calculating ppm values is rather complicted for Bruker format files. However, the following will give the correct output:

python

offset = (float(dic['acqus']['SW']) / 2) - (float(dic['acqus']['O1']) / float(dic['acqus']['BF1']))
start = float(dic['acqus']['SW']) - offset
end = -offset
step = float(dic['acqus']['SW']) / zero_fill_size

ppms = np.arange(start, end, -step)[:zero_fill_size]

We can now plot the spectra with the correct ppms.

python

fig = plotspectra(ppms, data)

Phase correction

The next step is to phase correct all the spectra. The package nmrglue provides a few automated algorithms that do a reasonable job with most normal, good quality spectra. Thankfully, that's what we have here.

python

for n in range(0, data.shape[0]):
    data[n, :] = ng.proc_autophase.autops(data[n,:], 'acme')
fig = plotspectra(ppms, data)

Automatic phase correction of NMR spectra

2014-06-01T15:36:00+00:00

This notebook demonstrates automatic phase correction algorithms implemented for nmrglue. Two standard algorithms are implemented:

ACME algorithm by Chen Li et al. Journal of Magnetic Resonance 158 (2002) 164-168
Naive peak minima minimisation

The outputs for the two algorithms are shown below. Automatic phase correction can be used through the addition of an autops function to the proc_base set alongside the algorithm name to employ for scoring of phase. Custom algorithms can be provided via the same parameter.

python

import nmrglue as ng

dic, data = ng.bruker.read("/Users/mxf793/Data/THPNH/Extract/1d/103/")
data = ng.bruker.remove_digital_filter(dic, data)

python

%matplotlib inline

python

import matplotlib.pyplot as plt
plt.figure(figsize=(10,4))
plt.plot(data)

python

/usr/local/lib/python2.7/site-packages/numpy/core/numeric.py:462: ComplexWarning: Casting complex values to real discards the imaginary part
  return array(a, dtype, copy=False, order=order)





[<matplotlib.lines.Line2D at 0x10cc7d090>]

python

data_fft = ng.proc_base.fft(data)
plt.figure(figsize=(10,4))
plt.plot(data_fft)

python

[<matplotlib.lines.Line2D at 0x10d0588d0>]

python

data_pc_pm = ng.proc_base.autops(data_fft,'peak_minima')

python

Optimization terminated successfully.
         Current function value: 0.000037
         Iterations: 90
         Function evaluations: 165

python

plt.figure(figsize=(10,4))
plt.plot(data_pc_pm)

python

[<matplotlib.lines.Line2D at 0x10d2e6710>]

python

data_pc_acme = ng.proc_base.autops(data_fft,'acme')

python

Warning: Maximum number of function evaluations has been exceeded.

python

plt.figure(figsize=(10,4))
plt.plot(data_pc_acme)

python

[<matplotlib.lines.Line2D at 0x10d4db850>]

NMRLab 1D NMR processing (MATLAB)

2014-05-28T11:14:00+00:00

This notebook uses a subset of the available processing features in NMRLab (+Metabolab) to process 1D NMR spectra. The output is saved as a CSV file that can be imported into pandas, PLS_Toolbox or any other package for subsequent analysis.

To use this workbook you will need an installation of MATLAB and a license for the NMRLab software (available free). To use segmental alignment (recommended) you will also need to install the icoshift package for MATLAB.

For more information on NMRLab/Metabolab see the following references:

U.L. Günther, C. Ludwig, H. Rüterjans NMRLAB - Advanced NMR Data Processing in Matlab. J Magn Reson, 145(2), 201-208 (2000)
C. Ludwig and U. Günther MetaboLab - advanced NMR data processing and analysis for metabolomics. BMC Bioinformatics, 12, 366, (2011)

Configure variables

Edit paths before for the data to process. The path should be the folder containing the experiment folders (1,2,3,4 etc.).

python

base_path = '/Users/mxf793/Data/THPNH/Extract/1d'
output_file = '/Users/mxf793/Data/THPNH/Extract/1d/processed.csv'
# include_match = []

We define a quick callback to pass widget values into global variables, allowing us to then pass these into %%matlab magic cells. We also define all the default config options used by widgets later, changes overwrite these defaults so widgets will not reset.

python

from IPython.html.widgets import interact, interactive, fixed
from IPython.html import widgets

def widget_callback(**kwargs):
    for k,v in kwargs.items():
        globals()[k] = v

spa_ref_spec_no=1
spa_max_shift=22

bc_algo=1
bc_npts=20
bc_tau=20
bc_par=6
bc_bas_noise=0

rea_ref_spec_no=1
rea_max_shift=22
rea_algo=1

bi_bin_size=0.05

Set up MATLAB connection

First we start up an instance of MATLAB server and connect to it - this will take a few moments. If it succeeds you will get a message returned MATLAB started and connected!.

python

%load_ext pymatbridge

python

Starting MATLAB on http://localhost:50397
 visit http://localhost:50397/exit.m to shut down same
....MATLAB started and connected!

Initialise NMRLab and global variables

We initialise MATLAB paths, and call the init to set up the environment for subsequent calls. We also pass in a number of variables from the Python environment, including the base_path for the data files, and a list of folders condtaining data ('fids') for processing.

python

%%matlab
NMRPAR.CURSET=[1,1];
nmrlab_init;
nmrlab_globals;

python

{Warning: The value of local variables may have been changed to match the
         globals.  Future versions of MATLAB will require that you declare
         a variable to be global before you use that variable.}
> In /Users/mxf793/MATLAB/Toolbox/nmrlab/nmrlab_init.p>nmrlab_init at 14
  In web_eval at 44
  In webserver at 241
*********************************************
*****    Welcome to  NMRLAB 3.5.0.0        ****
*********************************************

Setting global variables.
This computer is a MACI64
0.635
Running MATLAB version 7.1.
New swappath: /private/tmp/mxf793/15
Defining Paths for NMRLAB ...
Loading NMRDAT_EMPTY.

*********************************************

WAVLABPATH /Applications/MATLAB_R2011a_Student.app/wavelab does not exist.
The default path is MATLABHOME /wavelab
If you have installed wavelab elsewhere, set the WAVELETPATH
variable in nmrlab.m
WaveLab v .802 Setup Complete

python

import os

base_path = '/Users/mxf793/Data/THPNH/Extract/1d'
exclude = ['99999','9999','999']
folders = set(os.path.basename(folder) for folder, subfolders, files in os.walk(base_path) for file_ in files if file_=='fid' and os.path.basename(folder) not in exclude)
folders = list(folders)

# Sort folders (numerically)
folders.sort(key=int)

expfolders = folders
pathname_separator ='/'
nspc = len(expfolders)

python

%%matlab -i base_path,output_file,pathname_separator,expfolders
expfolders = str2cell(expfolders)

python

expfolders =
    '87'
    '89'
    '91'
    '93'
    '95'
    '97'
    '99'
    '101'
    '103'
    '105'
    '107'
    '109'
    '111'
    '113'
    '115'
    '117'
    '119'

python

%%matlab
%Dim 1
global h_DR

NMRPAR = struct();
NMRPAR.PATHNAMESEPARATOR = pathname_separator;

procSet = 1;
if(ishandle(h_DR))
    if(isfield(handles,'t_procSet'))
        set(handles.t_procSet,'visible','on');
        procSet = 1;
    end
end
pathBase{1} = base_path;
dataSet = [1 ];
dataType = 'B';
NMRPAR.CURSET = [dataSet(1),1];

python

%%matlab
expName = [];
for k = 1:length(expfolders)
    expName{1}{k} = expfolders{k};
end

Loading spectra FIDs

The next step is to load all the spectra FIDs into MATLAB for processing. This may take a moment.

python

%%matlab
expCounter = ones(length(dataSet),1);
for k = 1:length(expName)
    dataSet2 = find(dataSet(k)==dataSet);
    for l = 1:length(expName{k})
        pathname = [pathBase{k} NMRPAR.PATHNAMESEPARATOR expName{k}{l} NMRPAR.PATHNAMESEPARATOR];
        re1d2(pathname,dataSet(k),expCounter(k),pathname,dataType);
        expCounter(dataSet2) = expCounter(dataSet2) + 1;
        fclose('all');
    end
end

python

re(/Users/mxf793/Data/THPNH/Extract/1d/87/,1,1,/Users/mxf793/Data/THPNH/Extract/1d/87/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  D6 - 344}
re(/Users/mxf793/Data/THPNH/Extract/1d/89/,1,2,/Users/mxf793/Data/THPNH/Extract/1d/89/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  E6 - 345}
tline2 =
TopShimResult: 0.36
re(/Users/mxf793/Data/THPNH/Extract/1d/91/,1,3,/Users/mxf793/Data/THPNH/Extract/1d/91/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  F6 - 346}
tline2 =
TopShimResult: 0.29
re(/Users/mxf793/Data/THPNH/Extract/1d/93/,1,4,/Users/mxf793/Data/THPNH/Extract/1d/93/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  G6 - 347}
tline2 =
TopShimResult: 0.33
re(/Users/mxf793/Data/THPNH/Extract/1d/95/,1,5,/Users/mxf793/Data/THPNH/Extract/1d/95/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  H6 - 348}
tline2 =
TopShimResult: 0.39
re(/Users/mxf793/Data/THPNH/Extract/1d/97/,1,6,/Users/mxf793/Data/THPNH/Extract/1d/97/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  A7 - 349}
tline2 =
TopShimResult: 0.37
re(/Users/mxf793/Data/THPNH/Extract/1d/99/,1,7,/Users/mxf793/Data/THPNH/Extract/1d/99/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  B7 - 350}
tline2 =
TopShimResult: 0.38
re(/Users/mxf793/Data/THPNH/Extract/1d/101/,1,8,/Users/mxf793/Data/THPNH/Extract/1d/101/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  C7 - 351}
tline2 =
TopShimResult: 0.33
re(/Users/mxf793/Data/THPNH/Extract/1d/103/,1,9,/Users/mxf793/Data/THPNH/Extract/1d/103/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  D7 - 352}
tline2 =
TopShimResult: 0.37
re(/Users/mxf793/Data/THPNH/Extract/1d/105/,1,10,/Users/mxf793/Data/THPNH/Extract/1d/105/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  E7 - 353}
tline2 =
TopShimResult: 0.35
re(/Users/mxf793/Data/THPNH/Extract/1d/107/,1,11,/Users/mxf793/Data/THPNH/Extract/1d/107/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  F7 - 354}
tline2 =
TopShimResult: 0.39
re(/Users/mxf793/Data/THPNH/Extract/1d/109/,1,12,/Users/mxf793/Data/THPNH/Extract/1d/109/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  G7 - 355}
tline2 =
TopShimResult: 0.49
re(/Users/mxf793/Data/THPNH/Extract/1d/111/,1,13,/Users/mxf793/Data/THPNH/Extract/1d/111/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  H7 - 356}
tline2 =
TopShimResult: 0.35
re(/Users/mxf793/Data/THPNH/Extract/1d/113/,1,14,/Users/mxf793/Data/THPNH/Extract/1d/113/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  A8 - 357}
re(/Users/mxf793/Data/THPNH/Extract/1d/115/,1,15,/Users/mxf793/Data/THPNH/Extract/1d/115/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  B8 - 358}
tline2 =
TopShimResult: 0.63
re(/Users/mxf793/Data/THPNH/Extract/1d/117/,1,16,/Users/mxf793/Data/THPNH/Extract/1d/117/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  C8 - 359}
tline2 =
TopShimResult: 0.52
re(/Users/mxf793/Data/THPNH/Extract/1d/119/,1,17,/Users/mxf793/Data/THPNH/Extract/1d/119/,B)
0D dataset identified.
Reading acqus.
ans =
     0
ans =
     0
Parameters from BRUKER acqus files:

Update PROC and DISP
TD1 = 32768
SFO1 = 600.1328
SWH1 = 7194.2446
Reading FID ...
Real->complex conversion.
tline2 =

tline2 =
sy0207noesy H2O+D2O /opt/topspin3.0/data/youngsp/nmr youngsp {3  D8 - 360}
tline2 =
TopShimResult: 0.33

Importing Spectra and Pre-processing

The next step processes the input data automatically. Note that this will take quite some time if you're running with a large number of spectra at the same time. Run the cell and wait. Once the processing is completed a plot figure will be shown of all imported spectra.

python

%%matlab
phc0 = 0;
phc1 = 0;
dataSet2a = unique(dataSet);
for l = 1:length(dataSet2a)
    dataSet2 = dataSet2a(l);
    nspc = NMRDAT(dataSet2,1).ACQUS(1).NE;
    for k = 1:nspc
        fid_npts                        = length(NMRDAT(dataSet2,k).SER);
        zf                              = 2*2^nextpow2(fid_npts);
        swh                             = NMRDAT(dataSet2,k).ACQUS(1).SW_h;
        sfo1                            = NMRDAT(dataSet2,k).ACQUS(1).SFO1;
        NMRPAR.CURSET                   = [dataSet2,k];
        NMRDAT(dataSet2,k).ACQUS(1).DIM  = 1;
        NMRDAT(dataSet2,k).PROC(1).GIBBS = 1;
        NMRDAT(dataSet2,k).PROC(1).WDWF  = 'EM';
        NMRDAT(dataSet2,k).PROC(1).LB    = 0.3;
        NMRDAT(dataSet2,k).PROC(1).BC    = 32;
        NMRDAT(dataSet2,k).PROC(1).PH0   = phc0;
        NMRDAT(dataSet2,k).PROC(1).PH1   = phc1;
        NMRDAT(dataSet2,k).PROC(1).SMO   = 2;
        NMRDAT(dataSet2,k).PROC(1).SOL   = [32 32 0];
        NMRDAT(dataSet2,k).PROC(1).AUTOPHASE    = 1;
        NMRDAT(dataSet2,k).PROC(1).ZF    = zf;
        NMRDAT(dataSet2,k).PROC(1).REF   = [4.7458, zf/2, zf, swh, sfo1];
        NMRDAT(dataSet2,k).DISP.ax_typ   = 1;
        NMRDAT(dataSet2,k).DISP.PLOT     = 0;
        xfb(1,1);
        ref_tmsp;
        apk4;
    end
end
NMRPAR.CURSET = [dataSet(1),1];

% Set up some global vars
ppm = linspace(11,-1,32768);
nppm = size(ppm,2)
spc = [];

pppm = ppm;
pspc = spc;

% Output a figure showing all plots
clf;
hold on;

for k = 1:nspc
    spci = real( NMRDAT(1,k).MAT );
    spc = [spc; spci'];
    plot(ppm, spci);
end

spc1 = spc';

set(gca,'XDir','Reverse');

python

==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,1).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,1).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    2.3581
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,1).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 444 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,1).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 444 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,1).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 431.928 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,1).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 432.928 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,1).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 92.5369 -21.7569 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,1).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 80.1763 -8.0423 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,1).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 80.1763 -8.0423 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,2).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,2).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    1.8321
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,2).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 109 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,2).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 109 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,2).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 77.77 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,2).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 78.97 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,2).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 165.1121 -95.6301 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 78.97 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,2).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 106.7156 -30.8016 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,2).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 106.7156 -30.8016 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,3).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,3).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    1.9090
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,3).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 67 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,3).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 67 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,3).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 61.47 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,3).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 63.07 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,3).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 63.07 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,3).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 67.7947 -5.2467 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,3).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 67.7947 -5.2467 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,4).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,4).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    2.0172
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,4).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 75 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,4).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 75 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,4).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 69.312 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,4).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 70.312 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,4).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 81.6118 -12.5478 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,4).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 169.2485 -109.8635 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 81.6118 -12.5478 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,4).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 81.6118 -12.5478 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,5).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,5).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    1.7971
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,5).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 91 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,5).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 91 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,5).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 74.928 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,5).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 75.728 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,5).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 94.2839 -20.6039 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,5).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 270.909 -216.723 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 94.2839 -20.6039 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,5).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 94.2839 -20.6039 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,6).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,6).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    2.0382
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,6).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 88 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,6).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 88 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,6).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 83.596 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,6).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 84.796 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,6).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 95.7437 -12.1597 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,6).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 277.3022 -213.8182 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 95.7437 -12.1597 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,6).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 95.7437 -12.1597 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,7).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,7).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    1.8126
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,7).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 82 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,7).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 82 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,7).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 84.638 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,7).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 85.838 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,7).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 85.838 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,7).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 235.0558 -165.7318 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 85.838 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,7).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 85.838 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,8).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,8).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    2.5376
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,8).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 429 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,8).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 429 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,8).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 432.508 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,8).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 433.508 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,8).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 433.508 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,8).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 165.3203 -101.9283 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 433.508 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,8).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 433.508 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,9).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,9).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    1.8255
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,9).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 71 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,9).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 71 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,9).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 71.328 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,9).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 72.328 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,9).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 72.328 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,9).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 253.4552 -201.2272 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 72.328 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,9).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 72.328 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,10).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,10).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    2.2389
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,10).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 69 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,10).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 69 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,10).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 80.132 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,10).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 81.332 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,10).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 94.9834 -15.1694 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,10).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 213.8976 -147.3066 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 94.9834 -15.1694 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,10).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 94.9834 -15.1694 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,11).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,11).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    2.0196
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,11).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 79 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,11).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 79 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,11).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 79.508 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,11).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 80.708 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,11).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 96.8476 -17.9276 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,11).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 241.1291 -178.1931 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 96.8476 -17.9276 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,11).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 96.8476 -17.9276 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,12).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,12).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    3.1141
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,12).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 434 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,12).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 434 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,12).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 429.902 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,12).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 431.102 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,12).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 82.2845 -12.4125 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,12).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 204.9921 -148.6171 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 82.2845 -12.4125 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,12).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 82.2845 -12.4125 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,13).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,13).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    1.7759
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,13).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 71 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,13).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 71 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,13).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 59.218 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,13).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 60.418 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,13).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 60.418 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,13).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 92.2217 -35.3007 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 60.418 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,13).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 60.418 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,14).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,14).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    1.7551
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,14).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 434 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,14).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 434 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,14).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 421.032 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,14).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 421.832 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 421.032 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,14).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 77.6853 -18.4863 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,14).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 78.4757 -19.3637 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,14).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 78.4757 -19.3637 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,15).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,15).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    2.4110
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,15).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 445 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,15).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 445 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,15).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 442.384 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,15).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 442.784 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 442.384 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,15).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 97.8035 -17.1225 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,15).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 191.2078 -120.8428 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 97.8035 -17.1225 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,15).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 97.8035 -17.1225 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,16).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,16).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    2.0406
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,16).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 89 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,16).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 89 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,16).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 82.416 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,16).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 82.816 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,16).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 94.2529 -12.7009 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,16).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 221.6779 -154.2089 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 94.2529 -12.7009 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,16).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 94.2529 -12.7009 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,17).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,17).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 0 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
tEnd =
    2.7352
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,17).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 436 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,17).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 436 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
range =
    0.4000
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,17).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 442.222 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,17).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 444.022 0 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,17).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 96.4833 -13.8343 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,17).
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 220.3428 -151.3408 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
Avance 3 data.
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 96.4833 -13.8343 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
==============================================
XFB - Two-dimensional data processing.
XFB will be run for data set (1,17).
Avance 3 data.
SOL in DIM 1. SOL(32,32,0,76).
BC(32).
DIM 1 window function: EM, 0.3, 7194.2446, 76
GIBBS in DIM 1
ZF 32768 in DIM 1.
Avance 3 data.
DFT in DIM 1
Phase 96.4833 -13.8343 in DIM 1
Matrix transposed.
Processing DIM 1 done.
----------------------------------------------
You may want to delete your SER file now.
nppm =
       32768

Processing

To display the result of each step we define a simple MATLAB plot of mean and median spectra. This is usually enough to spot major issues in the spectra and see the improvements during processing. The output is captured and displayed using the ImageOut class we defined earlier.

The plot below shows the starting state of the spectra before processing.

python

%matlab pspc = spc1; pppm = ppm; spcmedian = median(pspc, 2); spcmean = mean(pspc, 2); clf; hold on; plot(pppm, spcmedian, 'k'); plot(pppm, spcmean, 'r'); set(gca,'XDir','Reverse');

Spectra alignment (TMSP)

We next apply an alignment on the TMSP standard to account for variation in the position of peaks arising during acquisition. Note that this does not account for differences in peak positions arising from pH (where peaks shift independently), this will be accounted for later using segmental shift (icoshift algorithm).

python

i = interact(widget_callback,
         spa_ref_spec_no=widgets.IntSliderWidget(min=1, max=nspc, step=1, value=spa_ref_spec_no, description="Reference spc:"),
         spa_max_shift=widgets.IntSliderWidget(min=1, max=100, step=1, value=spa_max_shift, description="Max shift"),
         )

python

%%matlab -i spa_ref_spec_no,spa_max_shift
[spc2, nul] = spcalign_tmsp(spc1, 1, 22, [150:155], 1);

pspc = spc2; pppm = ppm;
spcmedian = median(pspc, 2); spcmean = mean(pspc, 2); clf; hold on; plot(pppm, spcmedian, 'k'); plot(pppm, spcmean, 'r'); set(gca,'XDir','Reverse');

Baseline correction

Remove variability in the baseline using one of the pre-defined NMRLab methods. The defaults here use a spline baseline of 20 points along the spectra.

python

i = interact(widget_callback,
         bc_algo=widgets.DropdownWidget(values=[1,2,3,4,5,6,7,8], value=bc_algo, description="Algorithm"),
         bc_npts=widgets.IntSliderWidget(min=1, max=50, step=1, value=bc_npts, description="Reference spc:"),

         bc_tau=widgets.IntSliderWidget(min=1, max=50, step=1, value=bc_tau, description="Tau:"),
         bc_par=widgets.IntSliderWidget(min=1, max=10, step=1, value=bc_par, description="Poly order"),
         bc_bas_noise=widgets.FloatSliderWidget(min=1, max=10, step=0.01, value=bc_bas_noise, description="Baseline noise"),
         )

python

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)

<ipython-input-1-cb62b35ad0d7> in <module>()
----> 1 i = interact(widget_callback,
      2          bc_algo=widgets.DropdownWidget(values=[1,2,3,4,5,6,7,8], value=bc_algo, description="Algorithm"),
      3          bc_npts=widgets.IntSliderWidget(min=1, max=50, step=1, value=bc_npts, description="Reference spc:"),
      4
      5          bc_tau=widgets.IntSliderWidget(min=1, max=50, step=1, value=bc_tau, description="Tau:"),


NameError: name 'interact' is not defined

python

%%matlab -i bc_algo,bc_npts,bc_tau,bc_par,bc_bas_noise
[spc3, baseline, is_baseline] = baselinenl(spc2, bc_npts, double(bc_tau), double(bc_par), 0, bc_algo, bc_bas_noise);

spc3 = reshape(spc3, size(ppm,2), nspc);

pspc = spc3; pppm = ppm;
spcmedian = median(pspc, 2); spcmean = mean(pspc, 2); clf; hold on; plot(pppm, spcmedian, 'k'); plot(pppm, spcmean, 'r'); set(gca,'XDir','Reverse');

Scale spectra (TMSP)

python

%%matlab
spc4 = spcscale(spc3,1,1);

pspc = spc4; pppm = ppm;
spcmedian = median(pspc, 2); spcmean = mean(pspc, 2); clf; hold on; plot(pppm, spcmedian, 'k'); plot(pppm, spcmean, 'r'); set(gca,'XDir','Reverse');

Exclude regions (Water, TMSP, <ppm10)

We next remove regions from the spectra that are detrimental to subsequent analysis: first the water region in the middle of the spectra, then TMSP and finally <10ppm (extreme ends of spectra). These are removed so that noise variation does not interfere with scaling and multivariate analysis.

python

%%matlab -o ex_max,ex_1_end,ex_2_start,ex_2_end,ex_3_start
ex_max = size(ppm,2); ex_1_end = find(ppm<10, 1); ex_2_start = find(ppm<6, 1); ex_2_end = find(ppm<4.5, 1); ex_3_start = find(ppm<0.5, 1);

python

i = interact(widget_callback,
         # Default remove <10 ppm
         ex_1_end=widgets.FloatSliderWidget(min=1, max=int(ex_max), step=1, value=int(ex_1_end), description="Region (1) end"),
         # Default remove roughly ppm 4.5-6
         ex_2_start=widgets.FloatSliderWidget(min=1, max=int(ex_max), step=1, value=int(ex_2_start), description="Region (2) start"),
         ex_2_end=widgets.FloatSliderWidget(min=1, max=int(ex_max), step=1, value=int(ex_2_end), description="Region (2) end"),
         # Default remove TMSP
         ex_3_start=widgets.FloatSliderWidget(min=1, max=int(ex_max), step=1, value=int(ex_3_start), description="Region (3) start"),
         )

python

%%matlab -i ex_1_end,ex_2_start,ex_2_end,ex_3_start
excludes = zeros(size(ppm,2),1);
excludes(1:ex_1_end) = 1; excludes(ex_2_start:ex_2_end) = 1; excludes(ex_3_start:end) = 1;
excludes = logical(excludes);

spc5 = spcexclude(spc4,excludes,1);
ppme = spcexclude(ppm',excludes,1)';

pspc = spc5; pppm = ppme;
spcmedian = median(pspc, 2); spcmean = mean(pspc, 2); clf; hold on; plot(pppm, spcmedian, 'k'); plot(pppm, spcmean, 'r'); set(gca,'XDir','Reverse');

Realign NMR spectra

python

i = interact(widget_callback,
         rea_ref_spec_no=widgets.IntSliderWidget(min=1, max=nspc, step=1, value=rea_ref_spec_no, description="Reference spc:"),
         rea_max_shift=widgets.IntSliderWidget(min=1, max=100, step=1, value=rea_max_shift, description="Max shift"),
         rea_algo=widgets.DropdownWidget(values=[1,2], value=rea_algo, description="Algorithm"),
         )

python

%%matlab -i rea_ref_spec_no,rea_max_shift,rea_algo
spc6 = spcalign(spc5, rea_ref_spec_no, rea_max_shift, rea_algo);

pspc = spc6; pppm = ppme;
spcmedian = median(pspc, 2); spcmean = mean(pspc, 2); clf; hold on; plot(pppm, spcmedian, 'k'); plot(pppm, spcmean, 'r'); set(gca,'XDir','Reverse');

python

Shifts applied to spectra:
1: 0.
2: 2.
3: 6.
4: 5.
5: 4.
6: 22.
7: 11.
8: 2.
9: 22.
10: 22.
11: 22.
12: 1.
13: 1.
14: 3.
15: 5.
16: 7.
17: 2.

Segmental alignment

Here we use the icoshift algorithm for segmental alignment (this is the same as used internally by NMRLab, but with a simpler interface to the function to call).

python

sa_xt = 'average'
sa_intt = 'number'
sa_intn = 50
sa_shiftt = 'f'
sa_shiftn = 50


i = interact(widget_callback,
         sa_xt=widgets.DropdownWidget(values=['average','median','max','average2'], value=sa_xt, description="Target:"),

         sa_intt=widgets.DropdownWidget(values=['whole','number'], value=sa_intt, description="Interval type:"),
         sa_intn=widgets.IntSliderWidget(min=1, max=200, step=1, value=sa_intn, description="Interval number:"),

         sa_shiftt=widgets.DropdownWidget(values=['f','b','n'], value=sa_shiftt, description="Max shift type:"),
         sa_shiftn=widgets.IntSliderWidget(min=1, max=200, step=1, value=sa_shiftn, description="Max shift number:"),
         )

python

%%matlab -i sa_xt,sa_intt,sa_intn,sa_shiftt,sa_shiftn

if strcmp(sa_intt, 'number')
    sa_intt = sa_intn;
end

if strcmp(sa_shiftt, 'n')
    sa_shiftt = sa_shiftn;
end

spc7 = icoshift(sa_xt,spc6',sa_intt, sa_shiftt)';

pspc = spc7; pppm = ppme;
spcmedian = median(pspc, 2); spcmean = mean(pspc, 2); clf; hold on; plot(pppm, spcmedian, 'k'); plot(pppm, spcmean, 'r'); set(gca,'XDir','Reverse');

python

 Fast automatic searching for the best "n" for each interval enabled

 Co-shifting interval no. 1 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 2 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 3 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 4 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 5 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 6 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 7 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 8 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 9 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 10 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 11 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 12 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 13 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 14 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 15 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 16 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 17 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 18 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 19 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 20 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 21 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 22 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 23 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 24 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 25 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 26 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 27 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 28 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 29 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 30 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 31 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 32 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 33 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 34 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 35 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 36 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 37 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 38 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 39 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 40 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 41 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 42 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 43 of 50...
 Best shift allowed for this interval = 438

 Co-shifting interval no. 44 of 50...
 Best shift allowed for this interval = 437

 Co-shifting interval no. 45 of 50...
 Best shift allowed for this interval = 437

 Co-shifting interval no. 46 of 50...
 Best shift allowed for this interval = 437

 Co-shifting interval no. 47 of 50...
 Best shift allowed for this interval = 437

 Co-shifting interval no. 48 of 50...
 Best shift allowed for this interval = 437

 Co-shifting interval no. 49 of 50...
 Best shift allowed for this interval = 437

 Co-shifting interval no. 50 of 50...
 Best shift allowed for this interval = 386

Binning (Bucket spectra)

Binning reduces the size of the dataset by grouping regions of the spectra together. This speeds up subsequent analysis, and reduces the impact of small peak shifts (assuming these occur within a single bin).

python

i = interact(widget_callback,
         bi_bin_size=widgets.FloatSliderWidget(min=0.01, max=1.0, step=0.01, value=bi_bin_size, description="Bin size (ppm)"),
         )

python

%%matlab -i bi_bin_size
step_size = (max(ppme) - min(ppme)) / size(ppme,2);
points = bi_bin_size / step_size;
points = round(points / 2);
spc8 = spcbucket(spc7, points);
ppmb = linspace(max(ppme), min(ppme), size(spc8,1));

pspc = spc8; pppm = ppmb;
spcmedian = median(pspc, 2); spcmean = mean(pspc, 2); clf; hold on; plot(pppm, spcmedian, 'k'); plot(pppm, spcmean, 'r'); set(gca,'XDir','Reverse');

python

{Warning: 12 points left!}
> In /Users/mxf793/MATLAB/Toolbox/nmrlab/metabolab/spcbucket.p>spcbucket at 54
  In web_eval at 44
  In webserver at 241

Scale spectra (TSA/PQN)

python

%%matlab
spc9 = spcscale(spc8,1,1); % TSA by default; PQN must calculate the scaling vector manually

pspc = spc9; pppm = ppmb;
spcmedian = median(pspc, 2); spcmean = mean(pspc, 2); clf; hold on; plot(pppm, spcmedian, 'k'); plot(pppm, spcmean, 'r'); set(gca,'XDir','Reverse');

Variance stabilisation (auto/pareto/glog)

python

i = interact(widget_callback,
         glogexp=widgets.IntSliderWidget(min=1, max=10, step=1, value=4, description="glog(exp)"),
         y0=widgets.FloatSliderWidget(min=0.0, max=10.0, step=0.1, value=0.0, description="y0"),
         )

python

%%matlab -i glogexp,y0
spc10 = glogtrans(spc9, 10.0^-double(glogexp), y0 );
pspc = spc10; pppm = ppmb;
spcmedian = median(pspc, 2); spcmean = mean(pspc, 2); clf; hold on; plot(pppm, spcmedian, 'k'); plot(pppm, spcmean, 'r'); set(gca,'XDir','Reverse');

Export to CSV

The data resulting from the analysis is now stored in spc10 as a 2D array (variables down, samples across). We have the current ppms in a seperate variable. So we combine the (append to the left) and then output direct to a CSV file from MATLAB.

python

%%matlab
output = [ppmb', spc10]';
writecsv(output,output_file);

MetaboHunter 1D NMR Identification (Python Interface) Demo

2014-05-25T21:07:00+00:00

MetaboHunter is a web service for automated assignment of 1D raw, bucketed or peak picked NMR spectra. Identification is performed in comparison to two publicly available databases (HMDB, MMCD) of NMR standard measurements. More information about the algorithm is available in the published paper:

Tulpan, D., Leger, S., Belliveau, L., Culf, A., Cuperlovic-Culf, M. (2011). MetaboHunter: semi-automatic identification of 1H-NMR metabolite spectra in complex mixtures. BMC Bioinformatics 2011, 12:400

I find the service useful to give a first-pass identification of metabolites from 1D spectra, which can subsequently be confirmed or combined with identification via other methods. I originally wrote a Python interface as a standalone script, then as a Pathomx plugin, and have now moved the code into a reusable Python module with some extra IPython goodness. The walkthrough below demonstrates using the service with standard settings, passing a numpy array of ppms and peak heights. There is also a demo of a simple IP[y] Notebook widget set that can be used to configure the request.

The module and source code is available via PyPi and Github.

Setup

The module is on PyPi and has no funky dependencies. You should be able to instal the metabohunter from the command line:

python

pip install metabohunter

To use the module simply import it. The main module object provides two useful things: a request function that performs the request to the MetaboHunter service and a IPyMetaboHunter which provides nice widgets for IPython Notebooks and a synchronized config dictionary that can be passed to requests.

python

import metabohunter as mh
import numpy as np
import os
os.environ['http_proxy'] = ''

Input format

To make a request to the MetaboHunter service you need to provide two lists (or 1D numpy arrays) of ppm values (the x axis scale on an NMR spectra) and peak heights (y axis). Here we create some dummy data using an 50-element axis of 0-10 in 0.2 increments, together with a 50-element series of peak heights generated randomly.

python

ppms = np.arange(0,10,0.2)
peaks = np.random.random(50)*10

python

ppms

python

array([ 0. ,  0.2,  0.4,  0.6,  0.8,  1. ,  1.2,  1.4,  1.6,  1.8,  2. ,
        2.2,  2.4,  2.6,  2.8,  3. ,  3.2,  3.4,  3.6,  3.8,  4. ,  4.2,
        4.4,  4.6,  4.8,  5. ,  5.2,  5.4,  5.6,  5.8,  6. ,  6.2,  6.4,
        6.6,  6.8,  7. ,  7.2,  7.4,  7.6,  7.8,  8. ,  8.2,  8.4,  8.6,
        8.8,  9. ,  9.2,  9.4,  9.6,  9.8])

python

peaks

python

array([ 8.31680605,  6.04419835,  6.89353176,  6.00962915,  4.41208152,
        3.2333172 ,  1.39946687,  6.4614129 ,  6.20912024,  0.06888817,
        7.42894489,  6.7128017 ,  0.79111548,  8.85208481,  4.9710428 ,
        4.95762437,  9.82106628,  3.3606115 ,  8.71282185,  9.6313281 ,
        5.1396787 ,  6.90228616,  4.12455523,  3.71683751,  1.77995641,
        1.87159547,  5.43813402,  6.26325801,  9.17281811,  2.507874  ,
        0.64188688,  5.03782693,  6.93223808,  8.59120112,  2.95107901,
        9.70824585,  1.30386675,  1.02667654,  2.46923911,  9.02715511,
        2.42110673,  5.2022395 ,  8.79650171,  7.06068795,  9.45386543,
        4.38466017,  0.22570328,  3.25368676,  0.63608104,  6.98335382])

Performing a request

The results are returned back in a list of the same length as the input array. Mapped metabolites are represented by their Human Metabolome Database (HMDB) identifier whereas unmapped peaks are represented by None.

python

hmdbs = mh.request(ppms,peaks)
hmdbs

python

[None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 'HMDB00766',
 None,
 'HMDB00210',
 'HMDB01919',
 'HMDB01919',
 None,
 None,
 'HMDB00210',
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 'HMDB00763',
 'HMDB00617',
 'HMDB00763',
 'HMDB00259',
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None]

To throw away the None's and get the ppm values for the mapped metabolites you can do something like:

python

[(ppm, hmdb) for ppm, hmdb in zip(ppms, hmdbs) if hmdb is not None]

python

[(2.0, 'HMDB00766'),
 (2.4000000000000004, 'HMDB00210'),
 (2.6000000000000001, 'HMDB01919'),
 (2.8000000000000003, 'HMDB01919'),
 (3.4000000000000004, 'HMDB00210'),
 (6.8000000000000007, 'HMDB00763'),
 (7.0, 'HMDB00617'),
 (7.2000000000000002, 'HMDB00763'),
 (7.4000000000000004, 'HMDB00259')]

IPython Candy

To make the metabohunter module a bit nicer to work with from within IP[y] Notebooks, the module provides a simple class for generating widgets to control settings. The class is initialised with the default settings for the request, however you can pass additional variables (any of the keyword arguments allowed for request).

python

mhi = mh.IPyMetaboHunter(confidence=0.1, tolerance=0.5)

Once the objet is created you can call .display() to render the widgets in the current cell. Any changes to the variables are stored back into the IPyMetaboHunter class object (here mhi) and available in subsequent calculations.

python

mhi.display()

python

mhi.settings

python

{'confidence': 0.1,
 'database': 'HMDB',
 'frequency': '600',
 'metabotype': 'All',
 'method': 'HighestNumberNeighbourhood',
 'noise': 0.0,
 'ph': 'ph7',
 'solvent': 'water',
 'tolerance': 0.5}

The widgets manager makes the keyword arguments for the request available via a kwargs property. To provide these to the request function as keyword arguments we just need to unfurl it into the function call using **. Try adjusting the parameters above and seeing how they affect the results when re-running the request.

python

mh.request(ppms,peaks,**mhi.kwargs)

python

[None,
 None,
 None,
 None,
 None,
 'HMDB00172',
 'HMDB00011',
 'HMDB00518',
 'HMDB00510',
 'HMDB00510',
 'HMDB00518',
 'HMDB00510',
 'HMDB01547',
 'HMDB01547',
 'HMDB00101',
 'HMDB00208',
 'HMDB00192',
 'HMDB00162',
 'HMDB00014',
 'HMDB00122',
 'HMDB01401',
 'HMDB00272',
 'HMDB00902',
 'HMDB00085',
 None,
 None,
 'HMDB00215',
 None,
 'HMDB00393',
 None,
 None,
 None,
 None,
 None,
 'HMDB01392',
 'HMDB00617',
 'HMDB00303',
 'HMDB01406',
 None,
 None,
 'HMDB00232',
 'HMDB00902',
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None]

Dot Blot Analysis with ImageJ

2014-05-18T21:10:00+00:00

Analyzing a dot blot in ImageJ by background subtraction and measuring the integrated density of each dot. This dot blot image is available in the File/Open Samples menu in ImageJ 1.33s or later.

Method

This method usually requires background correction of the image, which can be done using the Process/Subtract Background command.

See image for profile plots (Analyze/Plot Profile) of the first row of dots before and after background correction was done using the Subtract Background command with the rolling ball radius set to 25 pixels.

After correcting the background, enable "Integrated Density" in Analyze/Set Measurements, create a circular selection, drag it over the first dot, press "m" (Analyze/Measure), then repeat for the other 27 dots.

Notice how the image now has a black background? It was inverted (Edit/Invert) so background pixel values are near zero, which is required for correct calculation of integrated density. You can invert the lookup table (Image/Lookup Tables/Invert LUT) to restore the original appearance of the image. The "Use Inverting Lookup Table" option in Edit/Options/Image will invert the pixel data and invert the lookup table.

Measuring cell fluorescence using ImageJ

2014-05-18T21:10:00+00:00

Determining the level of cellular fluorescence from fluorescence microscopy images in ImageJ

Open ImageJ. Note: ImageJ may be freely downloaded from here
Select the cell of interest using any of the drawing/selection tools (i.e. rectangle, circle, polygon or freeform)

From the Analyze menu select “set measurements”. Make sure you have area integrated intensity and mean grey value selected (the rest can be ignored).
Now select “Measure” from the analyze menu. You should now see a popup box with a stack of values for that first cell.
Now go and select a region next to your cell that has no fluorescence, this will be your background.

Repeat this step for the other cells in the field of view that you want to measure.

Size is not important. If you want to be super accurate here take 3+ selections from around the cell.

Once you have finished, select all the data in the Results window and copy and paste into a new spreadsheet (or similar program)
Use this formula to calculate the corrected total cell fluorescence (CTCF).

python

CTCF = Integrated Density – (Area of selected cell  X Mean fluorescence of background readings)

Make a graph and your done.

Notice that rounded up mitotic cells appear to have a much higher level of staining due to its smaller size concentrating the staining in a smaller space. If you used the raw integrated density you would have data suggesting that the flattened cell has less staining then the rounded up one, when in reality they have a similar level of fluorescence.

Cytoscape v3.0.2 on Mac OS X

2013-08-08T10:45:00+00:00

Cytoscape 3.0.2 was released on 1st August, a bugfix release for the 3.x series. However, there is a compatibility issue with the latest Java update from Apple resulting in Cytoscape hanging on startup. I'd followed advice found elsewhere to update to the latest Java version with no success, until I found this post from Tim Hull, outlining a simple way to check what version you have installed.

/usr/libexec/java_home -v 1.6 -exec java -version

If the output contains “xM4508”, you have the broken version. If it contains "xM4509", this is probably a different issue...

If you find that you have the bad version, you can download the updated from Apple using the links below:

The updated Java fixed the problem for me.

cx_Freeze and PySide on Mac

2013-04-18T09:00:00+00:00

I'd had success using py2app for building Mac binaries for distribution but wanted to give cx_Freeze a go since it's cross platform - allowing builds for Windows, Linux, and more. Unfortunately, attempting to build using cx_Freeze was resulting in errors:

python

libpyside-python2.7.1.1.dylib: No such file or directory

If found a tip suggesting you can do a touch libpyside-python2.7.1.1.dylib in you're applications folder to get rid of this error by creating a dummy file. But then you just end up with this instead:

python

copying Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/               PyQt4/Qt.so -> build/exe.macosx-10.6-intel-2.7/PyQt4.Qt.so
copying /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/QtCore.so -> build/exe.macosx-10.6-intel-2.7/PyQt4.QtCore.so
copying QtCore.framework/Versions/4/QtCore -> build/exe.macosx-10.6-intel-2.7/QtCore error: QtCore.framework/Versions/4/QtCore: No such file or directory

It's looking for the file in the wrong place: QtCore should be prefixed with the same path as PyQt4 before it. There are various version of PySide available - but whether you try to pip install or git clone https://github.com/PySide/pyside-setup and then python setup.py install you'll find that you end up in much the same place. In most cases however the installation simply fails.

Homebrew

So try a different tack: Homebrew. Homebrew is a package manager for Mac, that has a large number of successfully buildable software packaged specifically for the Mac. I've successfully used it to install a number of things, but steered clear of it for Python things that could be pip installed. However, there seemed to be little alternative - and actually I think brewed Python has it's advantages (being more up to date being one, being guaranteed to work with other brewed packages the other).

At this point my default Python install was the Mac default (Darwin, llvm-compat) version, installed under /Library/Python/2.7/ so first that needed to change.

Python

python

brew install python

The above will take quite a long time (particularly the Python bit) but you should now have a successfully brewed Python on which to base your apps. If you've had a different Python (e.g. system python) installed you may need to update your paths to ensure that the brewed one is being used - Homebrew actually gives you the instructions to correctly do this: but just add /usr/local/lib/python2.7/site-packages:/usr/local/share/python to your PATH variable in .bash_profile e.g.

python

export PATH=/usr/local/lib/python2.7/site-packages:/usr/local/share/python:$PATH

Pyside

An important fact about brewed Python is that it will continue to look in your system site-packages folder, and do so in preference to your brewed site-packages. If you have attempted to install Pyside (or anything else) before you may wish to remove them to stop conflicts.

Next install PySide via brew (the pip install does not work).

python

brew install pyside

The install cx_Freeze via pip (it's not in brew, and the pip works fine).

python

pip install cx_Freeze

Finally, check PySide are cx_Freeze are set up correctly by starting a Python interactive shell and entering the following (you should get the same output):

python

>>> import PySide
>>> import cx_Freeze
>>> PySide.__version__
'1.1.2'
>>> cx_Freeze.version
'4.3.1'

You're now free to build your apps.

Update 12.01.2014

There are some more things that now no longer work (as the app I'm building has got more and more complicated). However, there are solutions to both.

First of all is using PyQt5 with cx_Freeze raises an issue with finding the qt-menu-nib file that cx_Freeze uses to determine if it is packaging Qt - this folder no longer exists on Qt5. However, you can pass any old folder to cx_Freeze on the command line and it won't matter e.g.

python

python setup.py bdist_mac --qt-menu-nib=/usr/local/Cellar/qt5/5.2.0-beta1/plugins/platforms/

Another issue, documented here occurs when importing SciPy optimize based modules. That link contains a bit of hack code to fix it.

For an in-depth guide to building Python GUIs with PyQt5 see my book, Create GUI Applications with Python & Qt5.

Pelicans on Webfaction

2013-04-07T09:00:00+00:00

As mentioned in the previous post, I recently migrated this site over to the very clever Pelican. Setting it up was relatively straightforward using a combination of the official docs, this post and linked github repo from Dominic Rodger. That said, there were a few things that I stumbled at and non-obvious decisions that I've documented below.

Setting up

Before starting you need to install the packages. Thankfully everything you need is available via the python packaging service PIP. Unfortunately PIP isn't installed by default on Webfaction so you'll need to get that set up as per the instructions from Webfaction. So SSH into your account and do the following:

PIP

Make sure the destination directory exists. Enter mkdir -p $HOME/lib/pythonX.Y, where X.Y is the Python version, and press Enter.
Install pip. Enter easy_install-X.Y pip, where X.Y is the version of Python you wish to use, and press Enter.

In my case X.Y was 2.7

Pelican & dependencies

I then manually installed these dependencies, I'm not sure if all are required (or already present on Webfaction) - give it a go and let me know in the comments.

python

pip install feedgenerator   # to generate the Atom feeds
pip install jinja2          # for templating support
pip install pygments        # for syntax highlighting
pip install docutils        # for supporting reStructuredText as an input format
pip install pytz            # for timezone definitions
pip install blinker         # an object-to-object and broadcast signaling system
pip install unidecode       # for ASCII transliterations of Unicode text

If you’re not using Python 2.7, you will also need to pip install argparse.

In the Pelican setup it lists the following two as optional, but you'll need the first to use markdown flavoured markup. The second makes your text output prettier, I installed it.

python

pip install markdown        # for supporting Markdown as an input format
pip install typogrify       # for typographical enhancements

Now with all that out the way, we can install pelican. As with the above, it's a simple case of entering at the command line:

python

pip install pelican

Webfaction

First things first, we need to create a Webfaction webapp to hold our site. In fact, we need 2: one for the source code and settings, one for the output. Only the second of these is actually served publically - and Webfaction allows for this by making it possible to have applications with no association website. You do this through the Webfaction panel.

If you want to, you can put the generator code in your home folder instead but I think that is a bit confusing.

You can also decide here how you want to name the two apps. Initially I had mfitzp holding the source, and mfitzp_public holding the generated output. However, it dawned on me that if I wanted more than one Pelican site (quite possible) I was going to end up with multiple dud apps scattered around. To prevent this I created two apps, one pelican that will hold any Pelican site settings repos in a subfolder, and mfitzp which is the public served version of this site.

The first, pelican, was created as 'Static only (no .htaccess)', the second mfitzp as 'Static/CGI/PHP-5.4'. This latter decision only really has an impact if you need to use URL rewriting, or other .htaccess directives further down the line. I did, so I did. If you like this can also be 'Static only (no .htaccess)' and it'll be served by Webfaction's potentially-faster nginx server.

Remember to associate the public one with an active domain under 'websites'. Make sure it's working (if you've just set it up DNS could hold you up here).

Back to black

Back in the SSH session you can now cd ~/webapps and see the two new created folders pelican and yoursitenamehere. Now change into your pelican folder with

python

cd pelican              # change into your pelican folder
mkdir yoursitenamehere  # create a directory to hold your site config
cd yoursitenamehere     # change into your site's folder

We can now use the Pelican command to initiate the setup:

python

pelican-quickstart

Answer the questions and Pelican will create your initial folder. The important things to remember are that your output folder, relative to the current location is ../../yoursitenamehere.

Github

Following in Dominic's footsteps I decided to use github for my content hosting. However, Pelican has a handy tool to generate your setup, so you'll need to initiate the repo on the Webfaction side and re-home it onto github. Bear with me.

The first step is to log into your Github account and create the new repo, calling it whatever you like. Do not add a README.md as we need the repo empty to do the next thing - if you mess it up, just delete and recreate. Github is fine with that. Once created make note of your repo's read & write git url (SSH).

If you haven't set up github with your SSH keys you'll need to do that now. Generate a key on your Webfaction host using ssh-keygen. At the SSH prompt enter less ~/.ssh/id_rsa.pub to see the generated public key and copy-paste it into your Github SSH settings. Hit q to exit less.

Back in the shell, initiaise the github repo and then re-home it to your github repo.

python

git init
git add .
git commit -a -m "Initial commit"
git remote add origin git@github.com:<github-username>/<github-repo-name>.git
git push -u origin master

Sorted. The repo for your site is setup, and the repo now things Github is it's origin, so it will pull down from it to get updates. This means you can now clone that repo yourself to another machine and push your changes up via github.

Getting going

To all intents and purposes you are now finished and ready to make a beautiful website. But there are a few final things to consider.

Folders

Firstly, folder structure. Things are 'expected' to be laid out as follows by the system, but it's not immediately clear - meaning it can get a bit confusing why certain content is not showing up.

python

:::text
/content
    /images
    /pages
    /extra
    /<category-1>
    /<category-2>
/theme

The folder images is automatically copied through to appear at /images on your live site. Similarly pages is a special folder that anything contained within will be created as a static page, will not appear in feeds and may appear on link lists in certain forms. The URL structure for articles and pages is different (see later). Finally, extra is an optional folder, but extremely useful for copying through completely static content to the a particular destination. For example, on this site I'm using the following in my pelicanconf.py to copy through a static robots.txt and .htaccess:

python

FILES_TO_COPY = ( ('extra/robots.txt', 'robots.txt'),
                  ('extra/.htaccess', '.htaccess') )

URLs

Pretty-urls are a nice thing to have. Unfortunately, by default Pelican gives you a lot of .html everywhere. The official docs suggest solving this by setting your URL/SAVE_AS directives to put files as index.html under folders for the slug, so for example posts/your-slug-here/index.html. This works, as you can now put the URL as posts/your-slug-here/ and hit the file. Pretty URLs no redirects or .htaccess.

However, it doesn't work if you want to use custom themed error pages generated by Pelican (you can of course manually create them and copy them in the FILES_TO_COPY above, but you will need to maintain them in line with the theme by hand). To do that you either need to access that all pages will have URLs like /about.html or use .htaccess. And if you're going to use .htacess redirects you may as well make it handle the pretty URLs too.

So to start with add the following to your pelicanconf.py file.

python

ARTICLE_URL = 'posts/{slug}'
ARTICLE_SAVE_AS = 'posts/{slug}.html'
PAGE_URL = '{slug}'
PAGE_SAVE_AS = '{slug}.html'
AUTHOR_URL = 'author/{slug}/'
AUTHOR_SAVE_AS = 'author/{slug}.html'
CATEGORY_URL = 'category/{slug}'
CATEGORY_SAVE_AS = 'category/{slug}.html'
TAG_URL = 'tag/{slug}'
TAG_SAVE_AS = 'tag/{slug}.html'

This creates a set of URL formats where articles are linked to at posts/your-slug-here (note, no trailing slash, all the cool kids do it) and saved as the same name but with .html extension. Now save the config and create a new file names .htaccess in your content/extra folder - this will hold your .htaccess config for the live site. Add the following:

python

:::text
Options +FollowSymLinks
RewriteEngine On

# Remove trailing slashes.
# e.g. example.com/foo/ will redirect to example.com/foo
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)/$ /$1 [R=permanent,QSA]

# Redirect to HTML if it exists.
# e.g. example.com/foo will display the contents of example.com/foo.html
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule ^(.+)$ $1.html [L,QSA]

ErrorDocument 404 /404.html
ErrorDocument 403 /403.html

What it does is explained in the comments for each directive. The first chunk removes trailing slashes. The second chunk checks what is requested is not an actual file (e.g. static file), not a directory, and that <filename>.html exists, before rewriting the request to return that. The final two are for our error files.

Save the file and you're good to go.

Miscellaneous `peliconf.py`

Here are some additional extras for peliconf.py you might want to try out. Since I installed it I also turned on Typogrify by adding the following:

python

TYPOGRIFY = True

Because of the new feed variables in 3.0 some old themes won't work. Add this to fix:

python

FEED_DOMAIN = SITEURL
FEED_ATOM = 'feeds/atom.xml'

Express yourself (hey, hey)

It's time to write someting (finally). We'll create a test post and our two error document pages. So, first, simply create a new text file and enter the following:

python

:::text
Date: 2013-04-07 09:00
Author: Your Name
Email: your.email@your.email.provider
Title: A test post
Slug: a-test-post
Tags: test,post

Hello!

Save it in the content folder with the name a-test-post.md. You can save it under a sub-folder (create it) to categorise the post if you like. Then create another empty text file and add the following:

python

:::text
Title: 404
Slug: 404
Status: hidden

Not found
=========

Nothing to see here, move along.

Save it under the content/pages folder with the name 404.md. Edit a similar file for 403 Forbidden errors and save as 403.md.

Pelicans are go

Let's commit and push this to github incase we lose all our hard work. At the command line, in your pelican content source folder pelican/yoursitenamehere enter the following:

python

git add .
git commit -a -m "Initial commit"
git push

Now lets generate a site. In the same folder enter the following to create the content:

python

make html

You can also use:

python

make publish

The only difference is the latter also uses the config settings in publishconf.py including the quite useful RELATIVE_URLS = False directive (uncomment it first).

If you now access your site from the web, you should see your post in all it's glorious creativitiy. If it hasn't worked, drop a note in the comments and I'll work it out with you.

Automatic for the (lazy) people

I'm lazy, so I don't really like doing the whole commit-push-pull-make thing. Thankfully it's relatively easy to automate the whole thing with a short shell script. Still on your SSH session to webfaction enter:

python

nano ~/auto_pelican_publish.sh

This starts a simple editor to create a shell script. Simply copy and paste the following into the file, editing the paths as appropriate:

python

#!/usr/bin/env bash

cd <path-to-your-pelican-content-folder> # e.g. ~/pelican/yoursitenamehere
git pull
make publish

Save by pressing Ctrl-X and then Y. That's it. Back at the command line type:

python

chmod +x ~/auto_pelican_publish.sh  # to make the file executable

Next to add this to cron. Back at the command line enter:

python

crontab -e

Add the following to the top of the file, replacing your username:

python

:::text
PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin:/home/<webfaction-username>/bin
*/15 * * * * ~/auto_publish_pelican.sh

Again Ctrl-X and Y to exit and save. This will run the auto-publish script every 15 minutes, pulling from your remote git repository and then publishing the files it finds. All you need to do is make your commits on your local machine, push them to github, and within 15 minutes they'll be on your site. You can adjust the timer by changing the 15 to another number, e.g. 30 means every half an hour.

Wrapping up

Pelican is a really neat bit of software, that gets you away from all the faffing of dynamic web stuff to just concentrating on the important thing - writing. Hopefully this guide makes setting yourself up on Webfaction a little easier. If you hit any stumbling blocks in spite of this let me know in the comments! Happy pelicaning - whatever that involves.

Speed up MATLAB on MacOS X

2013-03-24T12:03:00+00:00

On MacOS X 10.5 there is considerable slow-down in the MATLAB editor and other GUI elements. The issue is related to a change in the default Mac Java 2D rendering engine from Quartz2D (10.4) and Sun2D (10.5). This newer rendering engine improves performance for figure drawing, but other GUI operations are slower.

The temporary fix is to instruct MATLAB to use the old rendering engine (Quartz2D). This will speed up scrolling and most GUI operations, at the cost of reduced figure drawing speed.

Open a terminal and browse to to your Matlab app bundle, eg.

python

cd /Applications/MATLAB_R2011a_Student.app

Your MATLAB folder will likely be different to this. A quick way to find it is to use the tab key. At the command-line enter cd /Applications/MATLAB then hit <tab> and the name should autocomplete.

If you're on 2008a or later, you need to browse down to the maci folder.

python

cd bin/maci

or for 64 bit

python

cd bin/maci64

Again, typing cd bin/maci<tab> should get you there.

At the command line type nano java.opts to create a new empty file and open a command-line text editor. You can copy and paste into this window, so simply copy the text below and paste it onto a new line in the file.

python

-Dapple.awt.graphics.UseQuartz=true

To save hit Ctrl-X(exit) then press Y to save and Return to accept the filename

You can also optionally add -Dapple.laf.useScreenMenuBar=false to turn off the Mac top-of-screen menu bar. This appears to also slightly increase speed.

Restart MATLAB and you should see a speed improvement.

As this fix may reduce drawing speed of figures, you may want to switch off the workaround while doing a lot of figure drawing. You can do so by exiting MATLAB, renaming the java.opts file e.g. to java.opts.off with mv java.opts java.opts.off and restarting MATLAB. To re-enable simply quit, rename the file back to java.opts with mv java.opts.off java.opts and restart.

Customize directory colors

2012-03-10T09:03:00+00:00

You can use the command ls --color (or an alias) to show directories with colours for folders, files, links, etc. However, you may not realise these colours can be easily configured using bashrc and a configuration file.

Edit your .bashrc file (in your home directory) to include the following line:

python

alias lc="ls --color=always"

This will enable coloured listings on all uses of ls (to save you typing --colors. Save the file and in your terminal window enter source ~/.bashrc to reload your bash config. Try an ls to confirm that you have got colors working.

On some systems (including Mac) the bash configuration is stored in ~/.bash_profile instead. You have a lot of options for configuring the directory colours. They can be stored in

Shell variable LS_COLORS which can be set in .bashrc via export LS_COLORS="COLOR_CONFIG"
In the file /etc/DIR_COLORS (you will need to be root to configure and this is global for all users)
In the file pointed by the variable COLORS (can be in your home directory)

Color configuration is done through a special formatted string:

python

FILE-TYPE Attribute codes: Text color codes:Background color codes

FILE-TYPE: is file type like DIR (for directories)
Attribute codes:
    00=none
    01=bold
    04=underscore
    05=blink
    07=reverse
    08=concealed
Text color codes:
    30=black
    31=red
    32=green
    33=yellow
    34=blue
    35=magenta
    36=cyan
    37=white
Background color codes:
    40=black
    41=red
    42=green
    43=yellow
    44=blue
    45=magenta
    46=cyan
    47=white

For example DIR 01;34 gives you a bold blue directory.

So to change the configuration globally edit the /etc/DIR_COLORS file as follows:

python

sudo nano /etc/DIR_COLORS

Look for:

python

DIR 01;34 # default is Bold blue with black background

And change it to:

python

DIR 01;34;41 # NEW default is Bold blue with RED background

Using LS_COLORS (in your own .bashrc file) the format is slightly different:

python

LS_COLORS='di=1:fi=0:ln=31:pi=5:so=5:bd=5:cd=5:or=31:mi=0:ex=35:*.rpm=90'

Here the codes are as follows:

python

di = directory
fi = file
ln = symbolic link
pi = fifo file
so = socket file
bd = block (buffered) special file
cd = character (unbuffered) special file
or = symbolic link pointing to a non-existent file (orphan)
mi = non-existent file pointed to by a symbolic link (visible when you type ls -l)
ex = file which is executable (ie. has 'x' set in permissions).

0   = default colour
1   = bold
4   = underlined
5   = flashing text
7   = reverse field
31  = red
32  = green
33  = orange
34  = blue
35  = purple
36  = cyan
37  = grey
40  = black background
41  = red background
42  = green background
43  = orange background
44  = blue background
45  = purple background
46  = cyan background
47  = grey background
90  = dark grey
91  = light red
92  = light green
93  = yellow
94  = light blue
95  = light purple
96  = turquoise
100 = dark grey background
101 = light red background
102 = light green background
103 = yellow background
104 = light blue background
105 = light purple background
106 = turquoise background

Control memcached from the command line

2012-02-05T05:02:00+00:00

memcached is a general-purpose distributed memory caching system originally developed by Danga Interactive for LiveJournal but now used by many other sites. It is often used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source (such as a database or API) must be read. Here we describe the options available from the command line to control a memcached instance via unix socket or IP:port.

Memcached can be set up to either take control commands via an IP address and port or via a unix socket.

Starting up

When listening via IP:port by default memcached listens for connections on port 11211 and accepts connections from INADDR_ANY. To prevent possibly malicious access you may want to consider restricting connections to an internal or firewalled connection. Alternatively you can set memcached to only listen for connections on localhost - i.e. the server running the instance of memcached using 127.0.0.1.

To start up an instance of memcached listening on localhost port 11211 only use the following:

python

::sh
memcached -d -m memory -l 127.0.0.1

memory is the maximum number of megabytes of memory you want memcached to use.

If you want to specify a different port to listen on use the -p PORT option.

Alternatively you can use unix sockets to communicate directly with the memcached instance. In this case commands can only be sent from a login shell that has access to the socket file - for example if you ssh into the server hosting it.

To start up memcached using unix sockets you would use the following command:

python

::sh
memcached -d -m memory -s ~/memcached.sock

This will create a file named memcached.sock in your home directory to communicate through.

Memcache management

Memcached comes with a number of useful management commands. Of these the most frequently useful are flush_all and the stats commands including stats, stats items and stats slabs.

To send a command to memcached you can use nc (netcat). The general form for all commands is:

python

echo "command" | nc 127.0.0.1 11211

Or for socket connections:

python

echo "command" | nc -U ~/memcached.sock

To flush the entire contents of your memcached instance you could issue the flush_all command with one of the following:

python

echo "flush_all" | nc 127.0.0.1 11211 echo "flush_all" | nc -U ~/memcached.sock

Results in:

python

[user@web37 ~]$ echo "flush_all" | nc -U ~/memcached.sock OK

[user@web37 ~]$

Statistics and state

You can use the stats commands to get information about the state of your memcached instance. For example:

python

echo "stats" | nc 127.0.0.1 11211 echo "stats" | nc -U ~/memcached.sock

Will output a list of stats for the current instance. Additional information can be found using stats items to give total stats about items stored in the cached and stats slabs which provides more information including performance hints.

Install Haiku under Mac OS X with Virtualbox

2012-02-05T03:02:00+00:00

Haiku is an open source operating system currently in development that specifically targets personal computing. Inspired by the Be Operating System, development began in 2001, and the operating system became self-hosting in 2008 with the first alpha release in September 2009, the second in May 2010 and the third in June 2011. Here we describe how to run a Virtualbox hosted Haiku OS on Mac OS X.

This has been tested on Mac OS X 10.8 (Mountain Lion) MacBook Air. You might need to tweak some settings on other hardware/OS versions.

To run Haiku on top of Mac OS X first download and install Virtualbox.

Download the version of Virtualbox for Mac OS X and open up the .dmg. Double-click on Virtualbox.mpkg to start the installation.

While Virtualbox is installing download a copy of Haiku. The simplest way to get Haiku running in Virtualbox is to download the pre-built VM image. Download the .zip file and double-click to unpack to a folder. Put this somewhere safe: this will hold your Haiku install.

Start up Virtualbox and click to create a new virtual machine. There is a short wizard to set this up. Set the operating system to Other and version to Other/Unknown.

Set the memory according to your host computer but 512MB should be enough for a comfortable Haiku install.

On the Virtual Hard Disk ensure that Start-up disk is checked and select "Use existing hard disk". Click the folder icon to the right on the drop down and browse to the location on your downloaded and unpacked Haiku folder. Select the file named haiku-r1alpha3.vmdk (or similar on later releases).

The default settings of Virtualbox work perfectly fine. However, Haiku occasionally locks up on the boot loader screen. A simple fix is to enable dual processor support. To do this right click on the created virtual machine and choose settings. Navigate to System -> Processor and increase the processor(s) to 2. Click OK to save the configuration.

In order to make networking work from within Haiku you will also need to change the virtualised networking card to any of the Intel ones. In settings navigate to Network -> Adapter 1 -> Advanced (hidden drop down section) and change the adapter type for example to Intel PRO/1000 MT Desktop.

You can leave the attached to setting as NAT or alternatively bind it to a specific adapter in your host computer by choosing Bridged Adapter and then (for example) wlan0 for Wi-Fi/Airport.

Double click the virtual machine to start Haiku!

Your mouse and keyboard will be captured by Virtualbox/Haiku if you click in the window. To get it back for use in Mac OS X press the Command (Cmd) key.

Find all files containing a given string

2012-02-01T01:02:00+00:00

A quick one-liner to recursively search files for a given text string.

Open up a terminal session and enter the following - replacing "foo" with the text to search for.

python

find . -exec grep -l "foo" {} \;

You can also limit the search to files with a particular extension (e.g. HTML or CSS). The first form will only return files in the current directory

python

find *.html -exec grep -l "foo" {} \;

Alternatively you can search recursively while matching filenames using the -name parameter:

python

find . -name *.html -exec grep -l "foo" {} \;

You can pass other options to find including for example -mtime -1 to specify only files modified within the past day. You can also replace . with the name of any other folder to search:

python

find /var/www -name *.html -exec grep -l "foo" {} \;

This will search /var/www recursively for files named *.html and containing foo.

Martin Fitzpatrick - tutorials

Getting Started With Git and GitHub in Your Python Projects — Version-Controlling Your Python Projects With Git and GitHub

Installing and Setting Up Git

Understanding How Git Works

Version-Controlling a Project With Git: The Basics

Initializing a Git Repository

Checking the Status of Our Project

Tracking and Committing Changes

Using a .gitignore File to Skip Unneeded Files

Working With Branches in Git

Creating New Branches

Checking Out to a New Branch

Merging Two Branches Together

Deleting Unused Branches

Using a GUI Client for Git

Managing a Project With GitHub

Setting Up a Secure Connection to GitHub

Creating and Setting Up a GitHub Repository

Pushing a Local Git Repository to GitHub

Pulling Content From a GitHub Repository

Exploring Alternatives to GitHub

Conclusion

Working With Classes in Python — Understanding the Intricacies of Python Classes

Defining Classes in Python

Adding Class and Instance Аttributes

Providing Behavior With Methods

Writing Getter & Setter Methods

Writing Special Methods

Reusing Code With Inheritance

Wrapping Up Classes-Related Concepts

Conclusion

Getting started with VS Code for Python — Setting up a Development Environment for Python programming

Setup a Python environment

Setup Visual Studio Code

Usage and Configuration

Linting and Formatting Support (Optional)

Working With Virtual Environments

Understanding Workspaces in VS Code

Working With Git in VS Code (Optional)

Community-driven & open source alternatives

Conclusion

Using MicroPython and uploading libraries on Raspberry Pi Pico — Using rshell to upload custom code

Installing rshell

The rshell interface

Connecting to your Pico with rshell

Starting a REPL

Uploading a file

Using uploaded libraries

Auto-running Python

What's next?

3-axis Accelerometer-Gyro — Measuring acceleration and orientation with an MPU6050

MPU6050

Setting up

Reading values

Smoothing

Calibration

Dictionary Views & Set Operations — Working with dictionary view objects

Keys

All keys (set union)

Keys in common (set intersection)

Unique keys (set difference)

Unique keys from both (set symmetric difference)

Items

Merge (set union)

Common entries (set intersection)

Unique entries (set difference)

Unique entries from both (set symmetric difference)

Displaying images on OLED screens — Using 1-bpp images in MicroPython

Setting up

Image Processing

Example images

Portable Bitmap Format

Loading images

Displaying an image

Portable Graymap Format

Loading images

Packing bits

Packing script

Loading images

Animation

Using a `.gitignore` File to Skip Unneeded Files

All keys (`set` union)

Keys in common (`set` intersection)

Unique keys (`set` difference)

Unique keys from both (`set` symmetric difference)

Merge (`set` union)

Common entries (`set` intersection)

Unique entries (`set` difference)

Unique entries from both (`set` symmetric difference)