Alex T.

Bringing Maven and AWS CodeArtifact Together

Fri, 22 Mar 2024 00:00:00 +0000

As a part of migrating our infrastructure fo AWS, we use CodeArtifact for hosting and caching Maven dependencies as well as mirroring the external repositories. Setting up CodeArtifact repositories is not too difficult, but making it work with Maven in a local environemnt is very chalengging due to a lot of AWS security constrains. In this post, I would like to share some experience and ideas on harmonising Maven and CodeArtifact when setting up my working environment.

Let’s assume that AWS CLI v2 has been installed

$ aws --version
aws-cli/2.15.30 Python/3.11.8 Linux/6.5.0-26-generic exe/x86_64.ubuntu.23 prompt/off

and there is an AWS profile dev configured properly in ~/.aws/config. This is an example AWS profile using the new AWS SSO:

...
[profile dev]
sso_session = dev-sso
sso_account_id = 0123456789
sso_role_name = Developer
region = ap-southeast-2
output = json

[sso-session dev-sso]
sso_region = ap-southeast-2
sso_start_url = https://d-9876543210.awsapps.com/start

That is, if we test the account with aws sts get-caller-identity, it returns a good response

$ aws sts get-caller-identity --profile dev
{
    "UserId": ".....",
    "Account": "....",
    "Arn": "arn:aws:sts::..."
}

Then we need Apache Maven version 3.6 or later.

$ mvn --version
Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)

AWS CodeArtifact Repositories

Let’s also assume that, we set up a CodeArtifact repository with details can be found via AWS Web console, then a repository of the owner 0123456789 as summarized below:

Region: ap-southeast-2
Owner: 0123456789
Domain: company-repositories

So the repository URL would be https://company-repositories-0123456789.d.codeartifact.ap-southeast-2.amazonaws.com/maven/dev/.

Apache Maven Settings

By default, Maven will look for a file settings.xml in its installation directory ${maven.home}/conf/ or the current’s user home directory ~/.m2. You can also explicitly specify a settings file using mvn -s path/to/settings.xml but that’s another story. It’s safe to assume we use the ~/.m2/settings.xml file with the content as below (the XML namespaces are removed for readability)

<settings>
  <profiles>
    <profile>
      <id>default</id>
      <repositories>
        <repository>
          <id>dev-repo</id>
          <url>https://company-repositories-0123456789.d.codeartifact.ap-southeast-2.amazonaws.com/maven/dev/</url>
          <layout>default</layout>
        </repository>
      </repositories>
    </profile>
  </profiles>

 <!-- Authentication part -->
  <servers>
    <server>
      <id>dev-repo</id>
      <username>aws</username>
      <password>${env.CODEARTIFACT_AUTH_TOKEN}</password>
    </server>
  </servers>
  
  <activeProfiles>
    <activeProfile>default</activeProfile>
  </activeProfiles>
</settings>

These settings can also be taken from CodeArtifact instructions. Just open AWS Web console, go to CodeArtifact, choose the corresponding repository then “View connection instructions”. In the pop-up dialog, choose “Maven mvn” as package manager client, then copy Step 4, 5, and 6 (optional for mirroring) to the corresponding section of ~/.m2/settings.xml as above.

CodeArtifact Authentication

In order to fetch Maven resources from CodeArtifact, we must set up the <server> section, in which the <username>, by CodeArtifact convention, must be aws. The <password> element must contain a Base64 encoded authentication token generated by the following AWS CLI command:

aws codeartifact get-authorization-token \
  --domain <repo_domain> \
  --domain-owner <domain_owner_id> \
  --region <region> \
  --query authorizationToken \
  --output text

Using shell script syntax, we can create an environment variable CODEARTIFACT_AUTH_TOKEN, assign it with the token, and expose it to be used by Maven. Please note that, we need to specify the AWS profile dev, too.

export CODEARTIFACT_AUTH_TOKEN=$(aws codeartifact --profile dev get-authorization-token \
  --domain company-repositories \
  --domain-owner 0123456789 \
  --region ap-southeast-2 \
  --query authorizationToken \
  --output text)

After that, we can run any Maven commands and Maven will use the generated CODEARTIFACT_AUTH_TOKEN variable, respectively. Otherwise, you will see an error message with the code Unauthorized (401).

[WARNING] Could not transfer metadata ... from/to dev-repo (https://company-repositories-0123456789.d.codeartifact.ap-southeast-2.amazonaws.com/maven/dev/): status code: 401, reason phrase: Unauthorized (401)

Problems & Solutions

In summary, after completing the configuration, there are few steps to enable Maven access to CodeArtifact repository.

AWS authentication: This step is required when using AWS SSO (recommended). In case you use the legacy credentials-based configuration, this step can be skipped.

$ aws sso login --sso-session dev-session
Attempting to automatically open the SSO authorization page in your default browser.
...
Successfully logged into Start URL: https://d-9876543210.awsapps.com/start

Generate the token and assign to the environment variable CODEARTIFACT_AUTH_TOKEN

export CODEARTIFACT_AUTH_TOKEN=$(...)

Now we can run Maven commands, e.g. mvn install

There would be no problem at all if the CodeArtifact authentication token does not have an expiry time. A token is only valid, by default, for 12 hours (perhaps enough for a working day 😅). You would need a new token after that. It would not be a big issue if you work solely with the command line all the time. You just have to ensure AWS session is not expired and repeat Step 2 to get a new token.

However, it’s very common that we devs are often using some Java IDEs, such as Eclipse, IntelliJ IDEA for development and testing. The IDEs often bundle their own version of Apache Maven but still use the settings file ~/.m2/settings.xml if not specified otherwise.

Here comes the tricky part. How can an IDE pick up a valid environment variable CODEARTIFACT_AUTH_TOKEN to ensure their built-in Maven work with CodeArtifact? We can be creative and write a wrapper script that generates the token before launching our favourite IDE. The IDE will pick up the token at startup time. Unfortunately, there are no easy ways to feed a running process with a new value of that variable when it expires. That means we might have to close the IDE app and open it again 😢.

There might be some third-party add-ons, for instance, AWS CodeArtifact + Maven for IntelliJ IDEA, that can update ~/.m2/settings.xml with the token as shown below:

Using the plugin is very convenient when we are working a lot or solely with the IDE and do not have to leave or restart it to pick up a new token. It can be combined with the AWS Toolkit plugin to manage AWS SSO profiles so that you don’t have to leave the IDE for (re)authentication.

In case of a mixed working environment of IDE and Maven CLI, it becomes less convenient as you would have to turn to the IDE plugin to update the token. For instance, I don’t build with the IDE built-in Maven often as they are struggling with large Java projects. Most of my Java builds are done via Maven CLI. The IDE is merely for code editing and debugging.

Even worse in case the IDE Maven and Maven CLI share the same settings.xml, it won’t work well because the plugin will replace the part ${env.CODEARTIFACT_AUTH_TOKEN} in settings.xml with the actual token whilst Maven CLI expects <password>${env.CODEARTIFACT_AUTH_TOKEN}</password> to be intact.

There are two potential solutions for a mixed environment like that.

Create a wrapper that takes care of obtaining the token before actually executing Maven CLI. Then we can use it for both IDE and CLI. For instance, IntelliJ IDEA lets us choose our own Maven.
Create a script to obtain the token and replace the content of the element <password>...</password> in settings.xml.

The biggest advantage of these solutions is that you don’t have to rely on IDE plugins and they work well for both IDE’s Maven and CLI.

Maven CLI Wrapper

We can create a simple wrapper, namely, mvn-wrapper for Maven CLI as following.

#!/usr/bin/env bash
if ! command -v aws >/dev/null 2>&1; then
  echo "The command 'aws' not found, you must install AWS CLI v2."
  exit 1
fi
if ! env | grep -i '^AWS_ACCESS_KEY_ID' >/dev/null 2>&1; then
  if ! env | grep -i '^AWS_PROFILE' >/dev/null 2>&1; then
    echo "AWS credentials are not available and AWS_PROFILE is not set."
    echo "You must either login using 'aws sso login --profile <profile_name>'"
    echo "or use 'AWS_PROFILE=<profile_name> $(basename -- $0)'."
    exit 1
  fi
fi
echo "Generating CODEARTIFACT_AUTH_TOKEN ..."
export CODEARTIFACT_AUTH_TOKEN=$(aws codeartifact \
  get-authorization-token \
   --output text \
  --query authorizationToken \
  --domain "${DOMAIN}" \
  --domain-owner "${OWNER}")
if [[ -n "$CODEARTIFACT_AUTH_TOKEN" ]]; then
  mvn "$*"
else
  echo "Failed to generate CodeArtifact authentication token"
  exit 1
fi

The wrapper checks for the AWS CLI aws, then verifies whether the AWS access key or profile is provided. If everything is okay, it will generate the token and call the actuall Maven CLI with the arguments mvn "$*". Just to make sure it’s executable

chmod +x mvn-wrapper

Then, you can use it in place of mvn

$ DOMAIN='company-repositories' OWNER='0123456789' AWS_PROFILE=dev ./mvn-wrapper compile
Generating CODEARTIFACT_AUTH_TOKEN ...
[INFO] Scanning for projects...
[INFO] 
[INFO] -------------------< com.example:hello >--------------------
[INFO] Building hello 0.0.1
[INFO]   from pom.xml
[INFO] --------------------------------[ war ]---------------------------------
[INFO] 
[INFO] --- resources:3.3.1:resources (default-resources) @ hello ---
[INFO] Copying 1 resource from src/main/resources to target/classes
[INFO] 
[INFO] --- compiler:3.8.1:compile (default-compile) @ hello ---
[INFO] Nothing to compile - all classes are up to date
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.413 s
[INFO] Finished at: 2024-03-23T10:44:21+11:00

If you use Maven mostly with the aforementioned CodeArtifact repository, you can hard code the value of AWS_PROFILE, DOMAIN, and OWNER within mvn-wrapper in this way so that you don’t have to type a long command, just simply use mvn-wrapper install.

export AWS_PROFILE=${AWS_PROFILE:-'dev'}
export DOMAIN=${DOMAIN:-'company-repositories'}
export OWNER=${OWNER:-'0123456789'}

We then configure both IDE and Maven CLI to use the wrapper.

Updating Maven Settings

In this approach, we don’t create any wrapper for Maven CLI but update directly ~/.m2/settings.xml. This solution is inspired by the idea of the IntelliJ plugin mentioned above without any IDE. Let’s create a simple script update-maven-settings with the content as following:

#!/usr/bin/env bash
if ! command -v xmlstarlet >/dev/null 2>&1; then
  echo "The command 'xmlstarlet' not found, you must install it beforehand."
  exit 1
fi
if ! command -v aws >/dev/null 2>&1; then
  echo "The command 'aws' not found, you must install AWS CLI v2."
  exit 1
fi
if ! env | grep -i '^AWS_ACCESS_KEY_ID' >/dev/null 2>&1; then
  if ! env | grep -i '^AWS_PROFILE' >/dev/null 2>&1; then
    echo "AWS credentials are not available and AWS_PROFILE is not set."
    echo "You must either login using 'aws sso login --profile <profile_name>'"
    echo "or use 'AWS_PROFILE=<profile_name> $(basename -- $0)'."
    exit 1
  fi
fi
echo "Generating CODEARTIFACT_AUTH_TOKEN ..."
export CODEARTIFACT_AUTH_TOKEN=$(aws codeartifact \
  get-authorization-token \
   --output text \
  --query authorizationToken \
  --domain "${DOMAIN}" \
  --domain-owner "${OWNER}")
if [[ -n "$CODEARTIFACT_AUTH_TOKEN" ]]; then
  settings="${HOME}/.m2/settings.xml"
  echo "CODEARTIFACT_AUTH_TOKEN is set, start updating '${settings}'"
  xmlstarlet edit --inplace -u "_:settings/_:servers/_:server/_:password" -v "$CODEARTIFACT_AUTH_TOKEN" "${settings}"
  echo "Maven settings '${settings}', the element '<password>' has been updated."
  exit 0
else
  echo "Failed to generate CodeArtifact authentication token."
  exit 1
fi

After ensuring the script is executable with chmod +x update-maven-settings, we can just run it

$ DOMAIN='company-repositories' OWNER='0123456789' AWS_PROFILE=dev ./update-maven-settings
Generating CODEARTIFACT_AUTH_TOKEN ...
CODEARTIFACT_AUTH_TOKEN is set, start updating '/home/alext/.m2/settings.xml'
Maven settings '/home/alext/.m2/settings.xml', the element '<password>' has been updated.

If you check ~/.m2/settings.xml, you will see the updated token

<password>eyJ2ZXIiOjE....</password>

Again, if only working with the same repository, we can hard code the variables as with mvn-wrapper.

After updating settings.xml with the token, you can freely use Maven CLI or IDE until the token expires. Then you just need to re-run update-maven-settings. You can even go hard-core with setting up a cron-job that automatically runs the script update-maven-settings to update the token before it expires. Just ensure that the corresponding AWS profile is authenticated, though.

In this approach, we need an extra tool xmlstarlet to update XML files. The tool is often packaged by most of the common Linux distros but not readily installed. On a Debian/Ubuntu alike system, use

sudo apt install -y xmlstarlet

On a CentOS / RHEL / Amazon Linux system

# Use YUM
sudo yum -y install xmlstarlet
# or DNF
sudo dnf -y install xmlstarlet

On Mac OS, we might need Homebrew to install xmlstarlet

brew install xmlstarlet

On Windows, you can download the binary package and install it or use a package manager such as Chocolatey.

The second solution is my favourite and I have been using it for my daily development with Java and Maven. You can opt for either that fits well with your own working style and environment.

Happy Coding 💪

References

Building Node Canvas Container for AWS Lambda

Tue, 19 Oct 2021 11:36:00 +1000

The Cairo-backed node-canvas has been a pillar for many Node.js based projects that need to simulate a browser canvas on the server-side. One of our recent projects uses Apache ECharts to render some graphs. We have successfully shifted the rendering part of ECharts to the server side to ease the burden on the front-end Web browsers using node-canvas and jsdom.

Thing goes well until we decided to migrate the rendering task to AWS Lambda Node.js 14.x. Unfortunately, AWS Lambda runtime environment is quite slim and hence, insufficient for node-canvas. There are many issues reported on the project Github page. In summary, the root cause is that node-canvas must be built with native libraries when we run npm install. Unless you are running Amazon Linux 2, which is the native runtime environment for AWS Lambda functions, there will be discrepancy between your local environment with AWS Lambda runtime environment. The typical error message is Error: libXXX: cannot open shared object file: No such file or directory . I’m developing on a Ubuntu 20 LTS box but still got many of those errors.

After giving exhaustive trials and errors adding missing libraries, there is a final and ultimate error regarding the libz. Adding libz even with the exact version 1.2.9 into the Lambda layer does not make the error go away. I came across a post by Max Duncan and realised I haven’t thought about the other option with AWS Lambda, i.e. creating a container image. This approach has been introduced recently, around Dec 2020.

Using a similar approach, I can successfully build an AWS Node.js 14.x container for node-canvas for ECharts. Nevertheless, the same approach can be used to build working node-canvas containers for any other libraries as well.

Create a simple project as following

.
├── Dockerfile
├── index.js
└── package.json

The file index.jswill be the entry point with the handler.

'use strict';

exports.handler = async (event, context, callback) => {
  const result = {
    body: null,
    isBase64Encoded: false,
    statusCode: 200
  };
  console.log('Received a render request event');
  try {
    result.body = ... ; // call another function using Node Canvas
  } catch (error) {
    console.log(error);
    result.statusCode = 500;
    result.body = JSON.stringify(error);
    return callback(result);
  }
  return callback(null, result);
};

The package.json is a simple NPM specification with the dependency to canvas, for instance

{
  "name": "node-canvas-lambda-container",
  "version": "1.0.0",
  "dependencies": {
    "canvas": "^2.8.0"
  },
  "license": "MIT"
}

The Dockerfile is the Docker container specification in which we create a new container image based on AWS public Node.js 14 image.

FROM public.ecr.aws/lambda/nodejs:14

##
# Install necessary package for building Node.js Canvas
##
RUN yum -y update \
&& yum -y groupinstall "Development Tools" \
&& yum install -y nodejs gcc-c++ cairo-devel \
	libjpeg-turbo-devel pango-devel giflib-devel \ 
	zlib-devel librsvg2-devel

COPY *.js package* ./

RUN npm install

ENV LD_PRELOAD=/var/task/node_modules/canvas/build/Release/libz.so.1

RUN yum remove -y cairo-devel libjpeg-turbo-devel \
		pango-devel giflib-devel zlib-devel librsvg2-devel

# Set the CMD to your handler
CMD [ "index.handler" ]

Now we can build the container

docker build -t node-canvas-lambda .

If the build is successful, we can see the new Docker image in our local environment

$ docker images | grep node-canvas
node-canvas-lambda				latest			ae9f63e1981a   11 seconds ago   1.72GB

Start the container for testing (no need to deploy to AWS yet)

$ docker run -p 3000:8080 node-canvas-lambda
time="2021-10-19T01:27:46.528" level=info msg="exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)"

Per AWS documentation, the container includes the AWS runtime interface clients needed to run. We can test the Lambda locally with curl or similar tools. Note that, the port has to be exact Docker port mapping 3000:8080. The rest of the URL has to be 2015-03-31/functions/function/invocations.

Open another teminal (as we need to keep the Docker Node.js container running) and use the following command:

$ curl -X POST "http://localhost:3000/2015-03-31/functions/function/invocations" \ 
	--data '{}'

{"body":null,"isBase64Encoded":false,"statusCode":200}

If thing goes well, we should see some JSON outputs from the Lambda handler (i.e index.handler).

Chromium and macOS Environment Variables

Fri, 05 Jul 2019 12:33:36 +1000

Trying Chromium in macOS Mojave (10.14), I confronted with the popular issue regarding Google API key missing. After obtaining an API key along with client ID and secret, I put it on some shell startup scripts as following.

export GOOGLE_API_KEY="your_google_api_key"
export GOOGLE_DEFAULT_CLIENT_ID="your_client_id"
export GOOGLE_DEFAULT_CLIENT_SECRET="your_client_secret"

Even more careful, I also use launchctl setenv to ensure these variables are available for macOS GUI applications (usually launched via Finder, Dock, Spotlight or so).

[[ -x /bin/launchctl ]] && /bin/launchctl setenv GOOGLE_API_KEY $GOOGLE_API_KEY
[[ -x /bin/launchctl ]] && /bin/launchctl setenv GOOGLE_DEFAULT_CLIENT_ID $GOOGLE_DEFAULT_CLIENT_ID
[[ -x /bin/launchctl ]] && /bin/launchctl setenv GOOGLE_DEFAULT_CLIENT_SECRET $GOOGLE_DEFAULT_CLIENT_SECRET

Sadly, it didn’t work under Mojave. When launching Chrome from Finder or Dock, the warning is still there. It seems Apple has silently altered macOS a lot and many existing work-arounds stop working under Mojave.

There are also some working methods involving renaming the original executable binary /Applications/Chromium.app/Contents/MacOS/Chromium, creating a new script to launch Chrome in which the aforementioned variables are set properly (even leading to automated scripts like this). Personally, I don’t like these ways as they are rather intrusive and therefore less upgrade-proof (I guess future updates of Chromium will override these launching scripts and you must circumvent that) and only consider them last resort.

Fortunately, I have found an old StackOverflow’s topic with two methods that work perfectly fine for macOS Mojave and systems before that as you can see in that SO’s topic. Better then, they are much less intrusive and upgrade-friendly.

Step 1: Open the built-in app “Script Editor” in /Applications/Utilities/, enter the following content

do shell script "/bin/launchctl setenv GOOGLE_API_KEY your_google_api_key"
do shell script "/bin/launchctl setenv GOOGLE_DEFAULT_CLIENT_ID your_client_id"
do shell script "/bin/launchctl setenv GOOGLE_DEFAULT_CLIENT_SECRET your_client_secret"

and then save (⌘ + s) under the format “Application”. Behind the scene, it will create a new macOS application and wraps the script content inside.

Step 2: Go to “Settings” –> “Users and Groups” –> choose the tab “Login Items”, click + and either point to the newly created application or simply drag and drop it to the box above.

Step 3: Log out and log in again. Chromium should see the variables and shows no warning.

Method 2: Via LaunchAgents

Step 1: Create a file environments.plist inside the folder ~/Library/LaunchAgents with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>gui-environments</string>
  <key>ProgramArguments</key>
  <array>
    <string>sh</string>
    <string>-c</string>
    <string>
    /bin/launchctl setenv GOOGLE_API_KEY your_google_api_key
    /bin/launchctl setenv GOOGLE_DEFAULT_CLIENT_ID your_client_id
    /bin/launchctl setenv GOOGLE_DEFAULT_CLIENT_SECRET your_client_secret
    </string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

Step 2: Enable it with launchctl

launchctl load ~/Library/LaunchAgents/environments.plist

Step 3: There is no step 3, just launch Chromium and see that it should work without warning ;)

In this method, you don’t even need to log out or reboot. The variables are made available to launchctl after Step 2.

Conclusion

The aforementioned methods work flawlessly under macOS Mojave (10.14) and less intrusive with respect to the target applications. As a result, we can upgrade or change the applications without concerning about re-creating, updating or changing the launching scripts. The only caveat is that, the environment variables will be exposed to all applications, and hence, could lead to some security concerns (e.g. regarding Google keys and secrets).

In that same SO’s topic, there is also a slightly different way to elevate a bit more to make it work for the whole system instead of only the particular user. It involves in putting the main script where environment vars are set in /etc/environment and creating a launching agent in /Library/LaunchAgents/ along with another daemon service in /Library/LaunchDaemons/ to monitor the content of /etc/environment.

Lenovo Thinkpad T430s Supervisor Password Removal

Mon, 24 Jun 2019 10:50:24 +1000

After a while merely working with macOS and Linux, I have recently decided to have an additional Windows box for some Windows-specific tools and development. I could be able to secure a second-hand Lenovo Thinkpad T430s on eBay for around 130 AUD, which is quite a decent price with CPU i5-3320M (2.6GHz), 4GB RAM, 14in 1600x900 LCD and its battery could last about two hours.

I was away when the seller delivered the laptop and thus unable to check anything right after receiving. It turns out the seller (or his seller before that, as he claimed) had lost or could not retrieve the BIOS supervisor password. I only noticed this issue when I tried unsuccessfully to alter the BIOS to UEFI mode in order to install and dual boot Windows and Ubuntu in UEFI. Without the correct supervisor password, I am not allowed to make any meaningful changes in the BIOS.

You got supervisor password locked when you see this after pressing F1 to enter the BIOS at boot time.

You may just Enter with blank and can still get into BIOS but most of the options are grayed out to your dismay.

Searching around on the Internet, most of Lenovo forum experts will tell you that the only way is to get Lenovo replacing the mainboard due to security setting. Some will point to Joe’s KeyMaker kit that costs at least 113 USD (98 plus 15 USD for shipping and handling).

Fortunately, there are some workarounds like this Reddit thread, David Zou’s post, this SuperUser’s thread, or this post, and many YouTube’s video like here or there. After few trials, I could successfully reset the supervisor password in my T430s. Here I will summarize some basic and working steps with some important notes that might save you some money, time, and effort.

IMPORTANT

The following procedure involves some risky short-circuit of a mainboard’s component. I will bear no responsibility when you damage your own computers using these guidance. Proceed at your own risk.

The main idea is to locate and short-circuit the pins SDA and SCL of the EEPROM during the process of supervisor password checking.

Disassemble T430s to expose the EEPROM
Clear supervisor password by short-circuiting the EEPROM’s pins
- Turn on and keep short-circuiting the pins SDA and SCL of the EEPROM when entering the BIOS (pressing F1) and the BIOS is checking the password
- Enter the BIOS, change the supervisor password, and then save the changes (F10)

Disassemble T430s

The EEPROM is on the mainboard under the keyboard. Hence, we must remove the keyboard to expose it. Unless you are too familiar with opening a T430s, you should refer to the Hardware Maintenance Manual (either pdf or online version).

Using the instructions for “Removing and Installing the Keyboard - ThinkPad T430s, T430si”:

Remove the battery pack
Remove the memory cover
Locate and remove the two keyboard screws marked with red color.

Turn to the upper side, slightly press or push the keyboard a little towards the direction of the display to unlatch it.
- You better use a small flat screwdriver to gently push and lift up the keyboard
Pull the keyboard out gently to avoid breaking the thin keyboard cable.

Now we need to locate the EEPROM. As the chip is very small, it will take a little effort. It would be faster with a good loupe or similar. In my T430s, the chip is south of the CPU and hidden under the black sticky isolation tape.

According to the layout diagram shown by David Zou, the pins SCL and SDA are on the lower opposite side of the dot. The EEPROM in my laptop got some sticky from the isolation tape on the surface and therefore difficult to see the dot. In this case, you can use a pointy driver to gently scan all over the chip’s surface like I did.

The SDA and SCL pins are identified with two blue arrows. Note that the space between them is tiny, so make sure you are using appropriate tool to only short these pins and don’t touch anything else. In my case, I found it very precise and comfortable with a pointy driver from a cheap screwdriver set.

Clear Supervisor Password

After successfully locating the EEPROM and identifying the two pins, as David Zou wrote here, the steps and responses for BIOS and UEFI based computers are slightly different, please refer to his information in your particular case. As my T430s is UEFI based, the process is following.

Turn on the computer / laptop
Short-circuit the two pins SDA and SCL and keep them shorted while pressing F1 to enter BIOS setup. I was taken directly to the BIOS without any prompt or asking.
Stop short-circuiting after successfully entering the BIOS (otherwise, start over from Step 1)
Navigate to the menu Security > Password > Supervisor Password to change the supervisor password (please read the following note).

An important lesson I learned at this point (in a hard way) is that, you should change the supervisor password to something not blank. After clearing the supervisor password as above, I tested few times and put back every thing just to realize that the old or locked supervisor password kicked back in. That means I got locked out as before and must re-open the keyboard to start over the short-circuit.

The issue was somehow noticed by clocow in this Reddit thread and his suspect is due to the backup battery (the yellow one in the mainboard). He suggested to leave the battery disconnected for 3 hours and retried.

Nonetheless, I have just changed the supervisor password and experienced no kick back or reset so far. My suspect is that the reverting of old/locked supervisor password happens when the supervisor password is blank after the hiccup (i.e. the attempt to clear password via short-circuiting). That’s why I got the aforementioned simple solution.

Eventually, I am truly happy to gain control of my laptop without spending extra money and learn few more thing on the way.

Happy hacking (aka short-circuiting), mates !!!

P.S.

For other models, you should research carefully whether the supervisor password can be cleared using this method. In some models, the places to store passwords are totally different and you can risk burning or bricking a working laptop.
Information about the location of EEPROM and two pins of other Lenovo models can be found here or here.

References

Making JSP work with Spring Boot 2

Mon, 24 Dec 2018 00:00:00 +0000

One of my past projects relied heavily on XML based configurations and JSP technologies dated back to Spring Framework 3.x. It is still working fine after upgrading to Spring 5 and corresponding dependent libraries such as Hibernate 5, HikariCP 3 and Apache CXF 3.3.

Exploring further into Spring ecosystem, I decided to give a shot on migrating to Spring Boot 2 whilst keeping most of the existing MVC (e.g. controllers, validations, JSP views) working. Most of the significant information are scattered throughout the Internet and most of the useful tutorials are sort of “just-follow-these-steps-and-it-will-guarantee-to-work-but-nothing-will-be-explained”.

It took me much unexpected effort and trial-and-error to get things right not to mention numerous do’s and don’ts got undocumented or lost among a bunch of documentations. This post merely focuses some on major aspects on bringing Spring Boot 2 and Spring Web with JSP together. The versions that I consider here are Spring Boot 2.1 and respectively Spring 5.1.

Creating a Spring Boot Project

We can quickly create a new Spring Boot project using Spring Initializr web site or via command line or IDE (e.g. Eclipse, IntelliJ IDEA). For less verbose build configuration, I will opt for Gradle build instead of Maven.

For instance, with the “New Project” wizard with Spring Initialzr in Intellij IDEA, I picked “War” for “Packaging”. The other non-default option is “Gradle Config”, as mentioned above.

Project Configurations

Dependencies

Spring Boot 2 comes with a number of starters (conventionally named as spring-boot-starter-*) to help us integrating appropriate dependencies in our Spring projects. I pick the module “spring-boot-starter-web” that adds an embedded Tomcat servlet container and Spring MVC.

Note that the embedded Tomcat package does not include JSP by default, we must add the module “org.apache.tomcat.embed:tomcat-embed-jasper” as well. In case you need JavaServer Pages Standard Tag Library (JSTL), just add “javax.servlet:jstl”.

Apart from those, I also consider to use WebJars as it provides client-side web libraries in terms of JAR files. Thus, It’s truly convenient for Java based web projects to bundle popular front-end libraries such as jQuery, Bootstrap, Font-Awesome, to name but a few. I will illustrate the use of WebJars Bootstrap 4 module to our demo project.

Eventually, our main build configuration build.gradle should look like this.

buildscript {
  ext {
    springBootVersion = '2.1.1.RELEASE'
  }
  repositories {
    mavenCentral()
  }
  dependencies {
    classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
  }
}

apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'
apply plugin: 'war'

group = 'io.github.htr3n'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '1.8'

repositories {
  mavenCentral()
}

dependencies {
  implementation 'org.springframework.boot:spring-boot-starter-web'
  implementation 'org.webjars:bootstrap:4.1.3'
  providedRuntime 'javax.servlet:jstl:1.2'
  providedRuntime 'org.apache.tomcat.embed:tomcat-embed-jasper:9.0.14'
  testImplementation 'org.springframework.boot:spring-boot-starter-test'
}

You might notice that some Spring Boot modules such as spring-boot-starter-* do not have their version numbers. The plugin io.spring.dependency-management will automatically imports the spring-boot-dependencies bom with adequate versions for Spring Boot’s dependencies.

Auto-configurations

The aforementioned wizard created a DemoApplication class annotated with @SpringBootApplication to start our Spring Boot application. The @SpringBootApplication annotation includes three other annotations that perform several configuration tasks automatically, which are,

@SpringBootConfiguration embraces @Configuration (i.e. the annotated application class also provides Spring configurations)
@EnableAutoConfiguration will trigger Spring Boot auto-configuration mechanism
@ComponentScan will enable the scanning for Spring components annotated with @Component or its sub-types on the package of the main application.

It’s convenient but not mandatory to use @SpringBootApplication. For advanced configuring or fine-tuning, one can surely pick apart individual annotations and add/customise them to particular needs.

In order to create a deployable WAR file, Spring Boot team recommends to subclass SpringBootServletInitializer and override its configure() method.

package io.github.htr3n;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.web.servlet.support.SpringBootServletInitializer;

@SpringBootApplication
public class DemoApplication extends SpringBootServletInitializer {

  public static void main(String[] args) {
    SpringApplication.run(DemoApplication.class, args);
  }

  @Override
  protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
    return application.sources(DemoApplication.class);
  }
}

View Resolvers

In Spring Web MVC, the return value of a @Controller, if not a valid View object, should be resolved by a ViewResolver . Spring Boot 2, by default, will load InternalResourceViewResolver that combines with InternalResourceView for handling JSP based views. As such, a controller’s return values will be mapped to a JSP resource.

InternalResourceViewResolver, which is based on UrlBasedViewResolver, will use the following rule “prefix + view_name + suffix” for view resolution. For instance, if a view’s logical name is “index”, prefix = “/WEB-INF/jsp/” and suffix = “.jsp” then the resulting view will be “/WEB-INF/jsp/index.jsp”.

Therefore, we need to configure the prefix and suffix. This can be done quite easily in Spring Boot , either using application.properties

spring.mvc.view.prefix=/WEB-INF/jsp/
spring.mvc.view.suffix=.jsp

or an equivalent but longer Java based configuration

@Configuration
public class WebConfig implements WebMvcConfigurer {
  @Bean
  public ViewResolver jspViewResolver() {
    InternalResourceViewResolver viewResolver = new InternalResourceViewResolver();
    viewResolver.setPrefix("/WEB-INF/jsp/");
    viewResolver.setSuffix(".jsp");
    return viewResolver;
  }
}

Note that Spring Boot will map a logical view onto a JSP file inside the folder “src/main/webapp/WEB-INF/jsp/”.

IMPORTANTMany developers, especially who have a lot of experience with Spring Web MVC, tend to annotate the @Configuration class with @EnableWebMvc. It’s crucial to notice that @EnableWebMvc will switch off all default Spring Boot auto-configuration for Spring Web MVC. That means, JSP files and other resources might not be served correctly without extra configurations.

Embedded Servlet Container

By default, Spring Boot package spring-boot-starter-web includes Tomcat server via spring-boot-starter-tomcat which is rather sufficient. As we adopt Spring Boot’s convention for the embedded Tomcat, it will be later accessible at the default address “http://localhost:8080”. In case you want to fine tune Tomcat, please reference to the full documentation.

To illustrate how Spring Boot works with JSP technologies, I will use a simple login page that asks for an email and password, conducts some trivial validations, and then informs whether it’s successful or failed.

First, let’s create a simple JSP file “webapp/WEB-INF/jsp/login.jsp” as the login view.

<%@ taglib prefix="form" uri="http://www.springframework.org/tags/form" %>
<html lang="en">
<head>
  <link href="webjars/bootstrap/4.1.3/css/bootstrap.min.css" rel="stylesheet">
  <title>Login</title>
</head>
<body>
<div class="jumbotron">
  <h1>Spring Boot 2 + JSP Demo</h1>
</div>
<div class="container">
  <div class="row">
    <form:form method="POST" modelAttribute="loginForm">
      <div class="form-group">
        <label for="email">E-Mail</label>
        <form:input path="email" cssClass="form-control"/>
        <form:errors path="email" cssStyle="color: red"/>
      </div>
      <div class="form-group">
        <label for="password">Password</label>
        <form:password path="password" cssClass="form-control"/>
        <form:errors path="password" cssStyle="color: red"/>
      </div>
      <form:button class="btn btn-primary">Log in</form:button>
    </form:form>
  </div>
  <!--- check login status and display message -->
  <div class="row">
    <% Object status = session.getAttribute("loginStatus");
      if ("OK".equals(status)) { %>
    <div class="alert alert-success" role="alert">
      Congratulations! Login successfully.
    </div>
    <% }
      if ("FAILED".equals(status)) { %>
    <div class="alert alert-danger" role="alert">
      Login failed. Please try again!!!
    </div>
    <% } %>
  </div>
</div>
<script src="webjars/bootstrap/4.1.3/js/bootstrap.min.js"></script>
</body>
</html>

Nothing is spectacular here. I will use some tags from Spring form library to conveniently define and bind HTML form elements with the back-end controllers and model. The major elements are

<form:form method="POST" modelAttribute="loginForm">—indicates the HTTP method POST as well as a model attribute namely loginForm
<form:input path="email"> — the input element for email
<form:input path="password">—the input element for password
<form:errors .../> are placeholders for Spring’s validation errors (if any)

The later part of login.jsp is to check whether the login process succeeded or failed and display a corresponding message via access to a session attribute loginStatus . The actual value of loginStatus will be updated by the business logic of the controller. Other than that, the JSP view includes Bootstrap 4’s stylesheet and JavaScript as provided by WebJars and use Bootstrap 4 CSS classes for styling.

...
<link href="webjars/bootstrap/4.1.3/css/bootstrap.min.css" rel="stylesheet">
...
<script src="webjars/bootstrap/4.1.3/js/bootstrap.min.js"></script>

First of all, we will need a data object to transfer the login input data in the JSP view (e.g. email and password) with the controllers and models. Here comes the LoginForm class.

public class LoginForm {
  private String email;
	private String password;
  
  public LoginForm() {}
  public String getEmail() { return email; }
  public void setEmail(String email) { this.email = email; }
  public String getPassword() { return password; }
  public void setPassword(String password) { this.password = password; }
}

No worries about the annotations @NotBlank, @Email for now as they are part of data validation. Apart from those, LoginForm is just a normal bean with a no-arg constructor and the getters / setters.

Now, we create a LoginHandler class to coordinate and handle the whole login process.

@Controller
@RequestMapping("/login")
public class LoginHandler {
  private static final String LOGIN_VIEW = "login";
  private static final String LOGOUT_VIEW = "logout";
  // controller body  
}

The LoginHandler class is annotated with @Controller to indicate it is a web controller that will be automatically detected by Spring Boot. In the LoginHandler class, let’s define some handler methods.

Handling GET Requests

We create a showLoginView() method annotated with @GetMapping for handling GET requests, i.e. when a client opens the corresponding URL “http://localhost:8080/login”.

@GetMapping("/login")
public String showLoginView(Model model) {
  model.addAttribute("loginForm", new LoginForm());
  return LOGIN_VIEW;
}

This method simply adds a new model attribute loginForm (explained later in Data Binding) and returns a logical login view name “login” (which will be resolved to “login.jsp”).

Handling POST Requests

For processing POST requests (i.e. when the client enters some input data and submits the form), we define a login() method annotated with @PostMapping.

@PostMapping("/login")
public String login(@Valid @ModelAttribute("loginForm") LoginForm form,
                    BindingResult bindingResult,
                    HttpSession session) {
  if (!bindingResult.hasErrors()) {
    final boolean authenticated = "abc@test.com".equals(form.getEmail());
    final String loginStatus = authenticated ? "OK" : "FAILED";
    session.setAttribute("loginStatus", loginStatus);
  }
  return LOGIN_VIEW;
}

The input data of the submitted form will be explicitly mapped to the method parameter form via the annotation @ModelAttribute. The @Valid annotation and the parameter bindingResult are for data validation, and the session is for storing session data indicating a valid authentication.

The actual authentication business logic has been simplified so that an input email “abc@test.com” will pass. Of course, it should be revised with proper data and authentication process in real apps.

Handling Log-Out

For the sake of completeness, we should also add a handler method for logging out in which we remove the session attribute loginStatus added in the POST method login() and redirect to a “logout.jsp” page.

@RequestMapping("/logout")
public String logout(HttpSession session) {
  session.removeAttribute("loginStatus");
  return LOGOUT_VIEW;
}

Data Binding

We need to connect the HTML login form to a data transfer object (also known as command object, form object, form-backing object) of type LoginForm. Otherwise, Spring MVC will raise the infamous error “Caused by: java.lang.IllegalStateException: Neither BindingResult nor plain target object for bean name ’loginForm’ available as request attribute”.

There are more than one way to do that:

Declare a new method that returns a LoginForm object and is annotated with @ModelAttribute("loginForm"). Note that all @ModelAttribute annotated methods will be invoked before any handler methods. Thus, LoginForm objects can be fed with some initial data, for instance, from a database.
For the GET method showLoginView(), create a parameter of type LoginForm and annotate it with @ModelAttribute("loginForm")
Do the same as (2) but omit @ModelAttribute for the method parameter because according to Spring Web documentation, it will be resolved as @ModelAttribute implicitly
Create a parameter of type Model for the GET method showLoginView() and then invoke Model.addAttribute("loginForm", new LoginForm()) to add a model attribute loginForm

Even though I can just go with (3) for its simplicity, I always prefer the other three as they advocate explicit declaration. It might make the code a bit verbose but would be easy to understand and maintain later on. I exemplify (4) and implement in showLoginView().

Data Validation

In my previous Spring 3 projects, I used to create validators that implement the Validator interface and utilise ValidationUtils helper methods to check the input data. These validator beans will be injected into Spring controllers to validate the input before handing over to the actual business logics.

Nevertheless, we can also leverage Bean Validation 1.0 (JSR-303) and 1.1 (JSR-349) supported since Spring 4 and Bean Validation 2.0 (JSR-380) supported since Spring 5 via Hibernate Validator. These modules are already included with spring-boot-starter-web.

First, we annotate LoginForm’s fields with some Bean Validation constraints and corresponding messages.

public class LoginForm {
  
  @NotBlank(message = "E-Mail must not be empty.")
  @Email(message = "Please enter a valid e-mail address.")  
  private String email;

  @NotBlank(message = "Password must not be empty.")
	private String password;
	...
}

Then we need to update LoginHandler with respect to validation logics but mostly with the POST handler login(). The simplest way is to add a @Valid annotation to the parameter form. It indicates that Spring should pass form to a Validator beforehand. To get access to validation errors, we have to declare a parameter bindingResult of type BindingResult and place it immediately next to form. Inside login(), we can invoke the method BindingResult#hasErrors() to see whether there are some validation errors and act accordingly. In this case, the login process won’t proceed until no validation error occurs.

public String login(@Valid @ModelAttribute("loginForm") LoginForm form,
                    BindingResult bindingResult,
                    HttpSession session) {
  if (!bindingResult.hasErrors()) {
    ...
  }
}

Thanks to Spring tag “<form:errors .../>”, any validation errors will be displayed on the JSP login view. For instance, when I left both email and password empty, I got the following messages.

If the BindingResult parameter is not placed immediately next to the corresponding model attribute loginForm, Spring Boot will throw a BindException that will be resolved via DefaultHandlerExceptionResolver as an error HTTP 404 - Bad Request.

Building and Executing

The build.gradle created previously is mainly based on the Spring Boot Gradle plugin for dependency and task management.

./gradlew bootRun

When visiting http://localhost:8080/login, you can see the login page.

To build a WAR package that is executable and deployable

./gradlew bootWar

The resulting WAR file is “./build/libs/spring-boot-jsp-demo-0.0.1-SNAPSHOT.war”. It’s interesting that you can deploy the WAR file to a standalone Tomcat server as well as directly execute it with “java -jar” .

java -jar ./build/libs/spring-boot-jsp-demo-0.0.1-SNAPSHOT.war

In case you want to change the WAR file name, Spring Boot Gradle plugin inherits from, and therefore, offers similar options as, the official Gradle War plugin. For instance, I can get rid of the version and use the build project’s name to form the WAR file name by customise the task BootWar in build.gradle.

bootWar {
	archiveName "${project.name}.war"
}

Also note that, when applying together with Gradle war plugin, Spring Boot Gradle plugin will disable the task war. Should you want to use war alongside bootWar, just simply enable it.

war {
	enabled = true
}

Updates

The original Git repository with Spring Boot v2 is here: https://github.com/htr3n/spring-boot-jsp-demo
2024-03-23: I created a demo with Spring Boot v3 and JSP/JSTL at https://github.com/htr3n/springboot3-web-war. There are few changes, especially with jakarta.servlet instead of javax.servlet and upgrading WebJars Bootstrap v5.

References

Tour de Spring Boot: DI meets CoC

Tue, 04 Dec 2018 00:00:00 +0000

I started using Spring framework few years back in 2009 to develop demo systems for some EU projects. These projects featured a number of Web services some of which share back-end databases and some other talk to external services or WS-BPEL business processes.

The dependency injection implemented in Spring works so well and on top of that, many other parts like Spring MVC, data access, transactions messaging and even some third-party libraries such as Apache ActiveMQ, Apache CXF and JBoss (now Red Hat and likely IBM) Hibernate ORM, and many other helper libraries all came to play nicely.

One of my concerns is that configuring a complex Spring based project is truly tedious and error-prone. I have made countless trial-and errors and gone through lots of warning and errors, not to mention XML verbosity and complexity, Maven dependency hell, the conflicts or incompatibilities among numerous dependencies. Fast forward to 2018 I want to refresh my experience through existing Spring projects. I was thinking about similar rough paths when upgrading Spring and Hibernate to some prior versions until discovering Spring Boot.

To be precise, Spring Boot is not a magic wand that would help us to navigate through the forest of Spring framework complexity or automatically resolve certain problems. It is more or less some sort of convention over configuration that might reduce significant effort on making Spring applications quickly up and running as well as ensure better compatibilities among integrated libraries.

A Quick Start

Let’s consider the following Java code defining a REST controller for an endpoint “/” that shall display a greeting.

@RestController
class HelloWorld {
  
  @RequestMapping("/")
  public String hello(){
    return "Hello World!";
  }
}

Just with this very minimal code, we can have a working Spring application. Assume that the Spring Boot CLI is available, start our application with the following command.

spring run HelloWorld.java

Spring Boot will launch an embedded Tomcat server and initialise a Web application to serve the endpoint at “http://localhost:8080”. Visiting this address will give us “Hello World!”. What could be cooler than that !?

Now, let’s start a quick tour on some more appealing features that Spring Boot 2.1.0 offers.

Project Bootstrapping

Using Spring Initializr directly or via IDE /CLI, we can quickly bootstrap a dead simple or full-blown Spring development project. For instance, this command will create a Spring application development project, namely, hello-world, with JPA and H2 database.

spring init --dependencies=web,data-jpa,h2 hello-world

After asking some questions such as the project’s and package’s names, the command line will download the core package and create a new development project that is ready to be imported to any IDE. Other than that, Eclipse or IntelliJ IDEA can just walk you through simple wizards in which you can craft your own project and choose the necessary components for the new project.

Build and Dependency Management

Spring Boot 2 works with Spring Framework 5 and requires at least Java 8. You can either choose Maven or Gradle build system when creating the project. An excerpt of Maven pom.xml (stripped off few minor details) of our project is shown below.

<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>io.github.htr3n</groupId>
  <artifactId>hello-world</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>hello-world</name>
  <description>Demo project for Spring Boot</description>
    
  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.1.0.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
  </parent>
    
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <java.version>1.8</java.version>
  </properties>
    
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
      <groupId>com.h2database</groupId>
      <artifactId>h2</artifactId>
      <scope>runtime</scope>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
      </plugin>
    </plugins>
  </build>
</project>

A simplified pom.xml

This POM inherits many useful default settings and values from spring-boot-starter-parent. We might opt to not use spring-boot-starter-parent and go old-school to set everything up on our own. I have just recalled working with Spring 3 and trying to integrate an embedded Jetty server. I really struggled with different Jetty versions as well as other libraries to make them click. When I tried to upgrade the legacy projects to newer Spring and Hibernate versions, things all turned out really messy with loads of mysterious errors.

Spring Boot eases this kind of burden with some opinionated design and configuration decisions. We can truly leverage its strength and choose among a set of starters (beginning with spring-boot-starter-). The starters are intended for better compatiblity and smooth integration.

In the hello-world project, we have chosen three components spring-boot-starter-data-jpa (JPA), spring-boot-starter-web (Web), spring-boot-starter-test (for testing) along with mysql-connector-java (MySQL).

Autoconfiguration

Intelligent autoconfiguration is perhaps one of the most prominent and important parts of Spring Boot. There are different levels of configuring a Spring based application. The simplest form of autoconfiguration is the “Hello World” example mentioned above. The new Maven based project created via the wizard or command line consists of pom.xml along with application.properties, HelloWorldApplication.java and HelloWorldApplicationTest.java as shown in Figure 1.

package io.github.htr3n;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class HelloWorldApplication {

  public static void main(String[] args) {
    SpringApplication.run(HelloWorldApplication.class, args);
  }
}

The HelloWorldApplication class is annotated with @SpringBootApplication which is actually an umbrella for three annotations:

@EnableAutoConfiguration that triggers auto-configuration of Spring application context.
@ComponentScan that enables the scanning of classes annotated with @Component or its sub-types on the package where the application is located (that’s why it is often recommended to place the class annotated with @SpringBootApplication at the root package above other classes)
@SpringBootConfiguration is an alias to @Configuration allows for registering some more beans in the context or import additional configuration classes

The aforementioned annotations are sufficient to launch not-so-complex Spring applications. In case one wants to fine-tune the application context autoloading, individual annocations mentioned above and/or customised annotations can be leveraged.

Examining the Spring Boot auto-configure project on Github, we can see many scripts for essential libraries. Depending on what are detected on the classpath and the application-specific configuration directives, Spring Boot will automatically include and configure the corresponding components.

We can explicitly disable certain parts using @EnableAutoConfiguration(exclude={class1, class2, ...}) or the directive “spring.autoconfigure.exclude=class1, class2, ...” in application.properties.

App Launcher Configuration

The SpringApplication class is the main launcher of a Spring application. Prior to Spring Boot, in order to start a Spring app we have to manually initialise an ApplicationContext, parse system- and application-specific properties and input arguments (if needed), load all necessary beans and/or launch an embedded Web server like Tomcat or Jetty. Now, all of these tasks will be automatically done using SpringApplication with @SpringBootApplication (or a combination of its sub-annotations).

Apart from application.properties (or .yml), we can configure and customise a Spring app using a SpringApplication or SpringApplicationBuilder instance. For instance, the following code will turn off the banner when launching our Spring app

public static void main(String[] args) {
  SpringApplication app = new SpringApplication(HelloWorldApplication.class);
  app.setBannerMode(Banner.Mode.OFF);
  app.run(args);
}

which is equivalent to spring.main.banner-mode=off in application.properties. Spring Boot comes with numerous predefined properties. Indeed, almost every autoconfigured component offers some default property settings. These predefined settings will be overriden with our own settings, if exist.

Properties Loading

By default, Spring Boot will bind external properties from application.properties (or .yml) at runtime. That is called externalized configuration. More details on application properties can be found here.

Warning
Please beware of the trailing whitespaces in Spring .properties as Spring Boot won’t trim off any of them and in some cases that might lead to unexpected issues or even crash the applications (see my other relevant post)

Besides the default application.properties, Spring allows us to activate various profiles using the naming convention application-{profile}.properties. It comes very handy in case we look forward to separating testing-specific environments, for example, using application-test.properties, from production environments. We can still ask to load property sources with @PropertySource as usual in Spring.

Components and Beans Configuration

Other components and bean configurations can be loaded from different sources including XML files, Java classes (with @Configuration), or Groovy (also annotated with @Configuration as Java or via Groovy Beans DSL “beans {...}”).

Note that, all components in the sub-paths under the main class annotated with @SpringBootApplication will be automatically scanned and initialised. The rest of beans configuration will be handed over to Spring’s dependency injection mechanism.

Web Configuration

Spring Boot often tries to establish an appropriate ApplicationContext based on classpath scanning and app-specific configurations. For instance, to enable the traditional Spring Web MVC, I simply include the component spring-boot-starter-web in the POM as shown in the previous section. Spring Boot will automatically adds relevant components for resolving and serving Web pages and resources.

We can add extra components such as WebFlux—the reactive framework added in Spring 5+ (spring-boot-starter-webflux), Thymeleaf template engine (spring-boot-starter-thymeleaf), Spring Web services (spring-boot-starter-web-services) and WebSocket (spring-boot-starter-websocket).

Spring Web MVC is totally capable to handle most of HTTP requests via servlets or controllers like the simple Hello World example above. Should you need more advanced control of RESTful endpoints, just add either one of Spring HATEOAS (spring-boot-starter-hateoas), Jersey (spring-boot-starter-jersey) or Apache CXF.

By default, the Web starter will use Tomcat as the underlying embedded application server. When we opt for another server, we must explicitly exclude Tomcat and add either spring-boot-starter-jetty (Jetty) or spring-boot-starter-undertow (Undertow) in our POM.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
  <exclusions>
    <exclusion>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-tomcat</artifactId>
    </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-jetty</artifactId>
</dependency>

Many of the embedded server’s settings can be defined in application.properties such as the network address, port, sessions, and so on. These settings can also be done programmatically by defining bean implementing WebServerFactoryCustomizer that enables access to customise the underlying WebServerFactory. For instance, you can refer to more fine-tuned customisations for Jetty, Tomcat, or Undertow.

Following up our example on excluding Tomcat and adding Jetty, we can create a simple bean to change the Jetty server from the default port 8080 to 9999.

package io.github.htr3n;

import org.springframework.boot.web.server.ConfigurableWebServerFactory;
import org.springframework.boot.web.server.WebServerFactoryCustomizer;
import org.springframework.stereotype.Component;

@Component
public class JettyServerCustomizer implements WebServerFactoryCustomizer<ConfigurableWebServerFactory> {

  @Override
  public void customize(ConfigurableWebServerFactory factory) {
    factory.setPort(9999);
  }
}

Data Access Configuration

Going through various versions, Spring framework always provides considerable support and different abstraction levels for database access. In general, it often requires some configurations for data sources and sometimes the intermediate layers such as pooling or caching.

Spring Boot has taken further steps forward by auto-configuring in some popular circumstances, for instance, embedded databases, database pooling, to name but a few. What we need to do is to add appropriate dependencies. In the hello-world project, H2 database engine will be auto-configured by Spring Boot.

<dependencies>
  ...
  <dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
    <scope>runtime</scope>
  </dependency>
  ...
</dependencies>

Note that, including database dependencies alone won’t trigger any auto-configuration. It’s only when Spring Boot found some sort of data access components such as spring-boot-starter-jdbc, spring-boot-starter-data-jpa (which transitively embraces spring-jdbc) on the classpath.

Atop the data connection layer, Spring Boot takes the liberty of choosing other data access components. Note that the design decisions taken by Spring Boot are often based on some popular and widely-used conventions in Spring based application developments. The decisions for data access, like other components, can be overriden with our own information. For example, I can define my own H2 data source in application.properties.

spring.datasource.url=jdbc:h2:mem:helloword_db;DB_CLOSE_ON_EXIT=FALSE
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=sa

Next, Spring Boot will try to initialise the database. High level data access components such as Spring Data JPA, Hibernate can be customised via specific properties. Nonetheless, Spring Boot can look into the classpath for a schema definition script, namely, schema.sql, and a data manipulation script, namely, data.sql and execute them if found. Here is an example of these scripts for a simple database customer.

-- schema.sql
DROP TABLE customer;
CREATE TABLE customer(
  id integer not null auto_increment,
  name varchar(255) not null,
  email varchar (255) not null,
  primary key (id)
);
-- data.sql
INSERT INTO customer(name, email) VALUES ('Bob', 'bob@test.com');

Instead, you may indicate the initialising scripts for specific databases using the naming convention schema-${platform}.sql and data-${platform}.sql where platform is taken from spring.datasource.platform. This convention helps us to separate vendor-specific data creation and manipulation scripts.

Last but not least, Spring Boot classpath scanning mechanism may help us to entirely skip JPA’s persistence.xml as it will try to search for all classes annotated with @Entity, @Embeddable, or @MappedSuperclass in our application’s classpath.

Development Support

Testing

Testing is always a crucial part of any development process and Spring apps are no exception. That’s why Spring Boot provides spring-boot-starter-test that comprises several handy components such as JUnit, Spring Tests, Mockito, to name but a few.

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

@RunWith(SpringRunner.class)
@SpringBootTest
public class ` {  
  @Test
  public void contextLoads() {
  }
}

Consider inside the project hello-world, we have a simple class HelloWorldApplicationTests annotated with @SpringBootTest that detects and initialises Spring Boot’s application context and testing environment. We can fine tune the testing environment in a similar manner as production’s (i.e. with @Configuration, @ContextConfiguration along with @TestConfiguration).

We won’t delve into evey detail of testing support but you can find an extensive list of test auto-configurations for JDBC, JPA, Neo, Redis, REST, Web MVC, and WebFlux as well as a rich documentation of Spring Boot testing features here.

In one of my previous posts, we can see that the annotation @JdbcTest has simplified a lot of settings needed for loading up databases, testing CRUD app with automatic transactions. As such it truly helps me better focus on the actual testing logics.

Troubleshooting

In order to dig deeper or to resolve bugs for our Spring apps, we may need more detailed information on logs and traces. Just start the Spring application under analysis with the command line option"--debug", system property “-Ddebug”, or set “debug=true” in application.properties. Here is an example of enabling debugging using command line.

mvn spring-boot:run --debug

The “--debug” option will enable the DEBUG logging level for some selected core Spring components. Likewise, we can also enable TRACE logging level with the “--trace" or “trace=true” in application.properties.

Developer Tools

Spring Boot equips us—Spring app developers—with a set of nice tools in hope that our life might become less miserable, or so they said ;)

<dependencies>
  ...
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
    <optional>true</optional>
  </dependency>
  ...
</dependencies>

One of my favourie features provided by spring-boot-devtools is automatic restart. That is, a Spring app can automatically reload whenever any changes on the classpath are detected. As stated in the documentation, it works really well with LiveReload to instantly refresh the corresponding Web browsers. The automatic restart feature mimics JRebel but in a slower and less advanced manner using two classloaders: a base classloader for unchanged resources and a restart classloader for changable ones.

Finale

Most of the decisions taken in Spring Boot are opinionated and some might like or dislike, especially hard-core experienced developers who know deeply about Spring.

Fairly speaking, the way Spring Boot hides the underneath complexity turns out rather ideal for beginners. An inexperienced developer can start the journey with minimal effort and configuration to have a running application. After that, more and more advanced configurations or features can be gradually added or customised until we no longer rely on Spring Boot. Instead of struggling with complex settings, Spring app developers can now better concentrate on the core system business logics to be designed and implemented.

Having said that, a Spring lover like me would truly appreciate the ideas and massive efforts behind the Spring Boot project to achieve an integrated set of various in-house and third-party libraries working smoothly. I think Spring development teams have made some bold and good decisions. With simple starting points and open to a plethora of advanced features, configurations and customisations, even experienced teams and developers would greatly benefit from that with more focused system designs and better speed of development.

Spring Boot properties issues with trailing whitespaces

Tue, 27 Nov 2018 00:00:00 +0000

TL;DR:Spring Boot does not trim off the trailing whitespaces in *.properties that in turn might lead to some unexpected errors. As the Spring development team decides to not fix this issue, we can configure code editors to avoid some similar problems.

The fun …

Coming back working with Spring after few years, I was thrilled trying out Spring Boot 2 with its superb cool auto-configurations and many more. It’s a lot much more fun to work with than Spring 3 used in some of my previous projects.

, the problem …

When tinkering with Spring Boot, I found a half-funny half-annoying issue that made one of my Spring apps failed to launch. As the project is rather clumpy and full of experimenting code, here I will replicate this issue in a smaller and simpler scale.

Let’s create a small Spring Boot project with JPA and H2. We can either use Spring Initialzr, Spring Boot CLI, or IDE such as IntelliJ IDEA, Eclipse or NetBeans. To keep it simple, I use Spring Boot CLI that can be easily installed.

$ spring version
Spring CLI v2.1.0.RELEASE
$ spring init --dependencies=data-jpa,h2 spring-boot-trailing-spaces
Using service at https://start.spring.io
Project extracted to '/dev/spring-framework/spring-boot-trailing-spaces'

The newly created Spring application can be launched with Maven.

$ cd spring-boot-trailing-spaces
$ mvn spring-boot:run
...

There should be no problem as we have not made any changes. Now, open “src/main/resources/application.properties” with an editor, then copy and paste the following code. Promise me that you don’t change the pasted code ;)

spring.datasource.url=jdbc:h2:mem:testdb
spring.datasource.driverClassName=org.h2.Driver 
spring.datasource.username=sa
spring.datasource.password=

These lines are typical data source configuration of an embedded H2 database engine that you might have seen around or used many times. Unfortunately, this time they will crash the Spring application.

$ mvn spring-boot:run
...
2018-11-27 21:20:54.289 ERROR 92868 --- [           main] o.s.boot.SpringApplication               : Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.zaxxer.hikari.HikariDataSource]: Factory method 'dataSource' threw exception; nested exception is java.lang.IllegalStateException: Cannot load driver class: org.h2.Driver

Walking across the lengthy exception stack and error messages, the main culprit seems to be “Cannot load driver class: org.h2.Driver”.

Why is that?

, the root causes? …

The very first thing came to my mind is to check whether H2 jar is included properly in the classpath.

$ mvn dependency:tree
...
[INFO] +- com.h2database:h2:jar:1.4.197:runtime
...

Yup, it’s there!

You can try further on, like I did, to remove $HOME/.m2 and re-run Maven or even restart the machine (but the app still doesn’t work!!!).

I have searched around for this specific issue and found some similar or the same exception, for instance, here, here, here or here. None of the solutions there can fix it.

It took me nearly half an hour to figure out the root cause. That is the single trailing space at the end of org.h2.Driver. Here I intentionally added in order to replicate exactly the original problem in my experimenting project. And the fault was mine who just lazily copied and pasted these few lines of Spring Boot configurations. After I removed all trailing spaces, the error disappeared.

Voilà!

… and the solutions?!

No copy & paste please, you might say ;)

I would blame the Pivotal development team (just kidding ;) as they should have handled such a trivial issue better because trailing whitespaces, by default in most editors, are not always visible nor printable. Indeed, trailing whitespaces in Spring properties has driven many developers crazy here, here, here, here or here. Note that it’s for now official that the Spring Boot team won’t fix this because they cannot safely trim properties. So, it’s our responsibilities to take care of these whitespaces.

Sidenote:
Whitespaces can also become worst enemies to source code controls (but not all whitespaces are born equal, for instance, two continuous trailing spaces in some flavours of Markdown should be persisted as they are translated into HTML line-breaks <br>).

Fortunately, most of code editors somehow can help to remedy the aforementioned issue either with built-in features or extensions. There are two possible non-exclusive methods (which can be combined of course)

M1. Showing or highlighting whitespaces
M2. (Automatically) trimming off whitespaces

For the first method, we configure the editors to force hard-to-see non-printable characters like tabs and spaces become visible. For instance, the problematic example above is shown in Visual Studio Code with the extension Trailing Spaces helps to highlight the trailing whitespaces in red. This way might be a bit UI-intrusive as highlighted whitespaces everywhere can distract many of us. I personally don’t really like having the whitespaces and tabs messing up with my code structure.

The second method can be automatic or manual in which the editors (maybe with support from extensions) can remove the whitespaces during saving or via some editing commands. However, beware the issues with Markdown trailing whitespaces, we should use this option per language specific settings if possible.

We will examine few popular code editors (please add more of yours in the comment) regarding these two methods.

Atom

Either Preferences ▸ Show Invisibles or config.cson

"*":
  editor:
    ...
    showInvisibles: true

Eclipse
- (M1) From version 3.7+: Preferences ▸ General ▸ Editors ▸ Text Editors : Turn on “Show whitespace characters” and fine-tune the option via “configure visibility”
- (M2) It’s rather tedious in Eclipse as many people have already complained. It’s perhaps due to the nature of Eclipse as a platform for several editors and plugins working together. IMHO, Eclipse should offer a common option to remove trailing whitespaces for all editors in the same level and place as “Show whitespace characters” above. Each kind of editor then might offer specific and optimal ways for certain file types. Unfortunately, for now we have to configure for each file type editor or formatter. For instance, Java editor Preferences ▸ Java ▸ Editor ▸ Save Actions , check “Additional actions”, choose “Configure”. Next, in the “Code Organizing” tab check “Remove trailing whitespace”. Unfortunately, properties file in our example does not provide such options in Eclipse.
- (M2) You may want to use the plugin AnyEdit Tools to clean up trailing whitespaces for all file types: Preferences ▸ AnyEdit Tools ▸ Auto ▸ Convert ▸ Remove trailing whitespace
- (M2) A universal approach, also a little more difficult, is to use “Search & Replace” to search with regular expressions for all of "[\t ]+$" (i.e. more than one whitespaces and/or tabs, please note the whitespace between \t and ]) and replace them with "" (blank/empty).
IntelliJ IDEA
- (M1) Preferences ▸ Editor ▸ Appearance ▸ Show whitespaces or View ▸ Active Editor ▸ Show whitespaces
- (M2) Since 2017.2 and later, there is an option Preferences ▸ Editor ▸ General ▸ Strip trailing whitespaces on Save with “All”, “Modified Lines” or “None” but the IDE does not quite behave, i.e. I chose “All” but the trailing spaces are still there after saving. It’s rather lame and embarrassing for such a highly praised and expensive IDEs.
Sublime Text
- (M1) Preferences ▸ Preferences.sublime-settings—User : "draw_white_space": "all" (show all whitespaces) or "draw_white_space": "selection" (only show within the selection)
- (M2) Preferences ▸ Preferences.sublime-settings—User : "trim_trailing_white_space_on_save": false
- (M2) Use the plug-in Trimmer for manually cleaning up whitespaces
Visual Studio Code
- (M1) Either Preferences ▸ Settings ▸ Render Whitespace or settings.json ▸ "editor.renderWhitespace": "boundary" (to render whitespaces everywhere except single space between words) or "editor.renderWhitespace": "all" (to render all whitespaces)
- (M2) Either Preferences ▸ Settings ▸ Render Whitespace or settings.json ▸ "files.trimTrailingWhitespace": true
- (M1+2) I use the extension Trailing Spaces that even goes further with red colour highlighting of whitespaces and provides extra commands for (manually) trimming whitespaces.
Vim
- You can follow the tips on highlighting unwanted spaces and removing unwanted spaces, some other advanced tips on Vi and Vim SE, or using a plug-in that can highlight trailing whitespaces and provide commands to strip all of them
  - (M1+2) vim-trailing-whitespace highlighting and trimming with :FixWhitespace
  - (M1+2) ntpeters/vim-better-whitespace highlighting and trimming with :StripWhitespace
  - (M1) ShowTrailingWhitespace: mostly trimming whitespaces
  - (M2) DeleteTrailingWhitespace: removing trailing whitespaces with :DeleteTrailingWhitespace or automatically when the buffer is saved

For the sake of completeness, there are some editor-independent solutions too.

sed—a stream editor (credits to Tim Pote’s excellent SO answer)

For POSIX

find . -type f -name '*.properties' \
 -exec sed --in-place 's/[[:space:]]\+$//' {} \+

For macOS/BSD

find . -iname '*.properties' -type f \
 -exec sed -i '' 's/[[:space:]]\{1,\}$//' {} \+

Wow, quite a long post for such a trivial issue, eh?! Thanks for reading to the end and I hope that my wasted time and effort here would save some of yours in future.

Cheers!

Back to basics: Test-driven Spring JDBC

Tue, 20 Nov 2018 00:00:00 +0000

Spring is a popular heavy-weight framework for developing Java/Groovy based applications. Its rich libraries and software stack can cover from front-end to back-end development. In 2009 I used Spring Framework 3 to develop web services and MVC + Hibernate portal for a fictious loan approval process. Despite a tad steep learning-curve, I could manage to get the services and portal up and running and integrated with third-party libraries quite smoothly.

In my prevous projects, I mostly used Spring with Hibernate and/or JPA for higher abstraction level of data access. Coming back working with Spring after few years, I want to delve into lower layer of data access to better understand what behind the scene of Hibernate/JPA abstraction layers. This post is sort of my note-to-self on Spring and JDBC (Java Database Connectivity), especially on JdbcTemplate. Besides, it also reports a tricky case with retrieving data with JdbcTemplate.queryForXXX() employed by several on-line tutorials which was efficiently exposed by appropriate tests.

Background

JDBC defines standard application programming interface (API) based on that a client can access a database. Each database vendor often provides low-level vendor-specific database drivers based on JDBC predefined interfaces. JDBC is considered the lowest recommended abstraction level to work with databases in Java.

In short, a typical approach to database access using JDBC comprises these basic steps:

Obtain a Connection, e.g. via DriverManager or DataSource
Create an instance of type Statement or its sub-types such as CallableStatement, PreparedStatement
Use the aforementioned statement to execute database queries
Retrieve and process the ResultSet (if any)
Close the statement and release all resources (connection is AutoCloseable)

Along these steps, we should also handle any database exceptions as well. You can find more details on JDBC programming here. Here is a simple example of accessing databases using JDBC and try-with-resources.

DataSource ds = ...; // obtain a DataSource object
try (
  Connection conn = ds.getConnection();
  Statement stmt = conn.createStatement()) {
  try (ResultSet rs = smmt.executeQuery("SELECT * FROM customer")){
    while (rs.next()){
      // process the ResultSet
    }
  } catch (SQLException e2) {
    e2.printStackTrace();
  }
} catch (SQLException e1) {
  e1.printStackTrace();
}

Fortunately, Spring, via JdbcTemplate, ofters a higher level of abstraction on top of Java JDBC that would save us a lot of boiler plate code and enable smooth integration with the rest of Spring framework’s ecosystem. As such, we can leverage other parts of Spring farmework, for instance, the awesome Spring Boot, to automate lots of configuration effort.

A simple CRUD project with Spring JDBC

Nothing is better than a hand-on development project that demonstrate how Spring JDBC works. We can start with Spring Initialzr and Spring Boot to jump start and better concentrate on the main code instead of numerous dependencies and configurations.

There are a few ways to bootstrap a project with Spring Initialzr. Most of the popular Java IDE such as Eclipse or Intellij IDEA with Spring extensions can create a new project via Spring Initialzr. You can also achieve the same result from the web site https://start.spring.io or even using Spring Boot CLI.

Here I will use Intellij IDEA for just showing the necessary steps and dependencies. You can use any of the aforementioned methods to obtain the same result.

Step 1:

Create a new project and choose Spring Initialzr.

Step 2:

Fill in necessary project information, keep default values for project’s type (Maven), language (Java), packaging (Jar), Java version (8).

Step 3:

Choose the section SQL and make sure the checkboxes of H2 and JDBC are ticked.

Step 4:

Finish the project creation wizard.

After Step 4, Intellij IDEA will create a new Maven Spring Boot project with some initial source and configuration files.

By default, the main configuration file for Spring Boot is application.properties.

Defining Domain Entities

We might not need a complex domain model but rather a simple entity mapped to a database table. For example, a Customer entity as shown in Java code and a corresponding customer table.

package io.github.htr3n.springjdbcsimple.entity;

public class Customer {
  private Integer id;
  private String name;
  private String email;

  public Integer getId() {
    return id;
  }
  public void setId(Integer id) {
    this.id = id;
  }
  public String getName() {
    return name;
  }
  public void setName(String name) {
    this.name = name;
  }
  public String getEmail() {
    return email;
  }
  public void setEmail(String email) {
    this.email = email;
  }
}

Configuring Database Connection

Normally, we have to provide sufficient information to establish database connections, for instance, JDBC url, authentication, and maybe some extra parameters. Per convention, Spring Boot can scan for data sources configurations as @Bean or in application.properties.

For better concentraton on test-driven JDBC CRUD, we can leverage @JdbcTest explained in the subsequent section to set up an in-memory embedded database and skip the details on data source configuration for now.

However, we must still initialise the database (otherwise Spring will complain that the table customer does not exist when our application or a test starts). By convention documented here, we simply add a file schema.sql in the folder src/main/resources which contains a simple SQL DDL script for creating a table customer.

DROP TABLE customer;
CREATE TABLE customer(
  id integer not null auto_increment,
  name varchar(255) not null,
  email varchar (255) not null,
  primary key (id)
);

Test-Driven CRUD

At the heart of our JDBC project is a CustomerDao.java with basic CRUD methods. The class CustomerDao is annotated with @Repository to indicate this is a data access component that is autodetected through normal Spring’s classpath scanning.

As JdbcTemplate will be used to work with H2 database, we just declared an @Autowired field, the rest will be taken care by Spring.

package io.github.htr3n.springjdbcsimple.dao;

import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;

@Repository
public class CustomerDao {
  @Autowired
  private JdbcTemplate jdbcTemplate;

  // Create
  public Customer create(Customer customer) { ... }
  // Retrieve
  public List<Customer> findAll() {...}
  // Retrieve
  public Optional<Customer> findById(Integer id) { ... }
  // Update
  public boolean update(Customer customer) { ... }
  // Delete
  public boolean delete(Integer id) { ... }
}

CRUD Tests

Before starting implementing the CRUD methods, we create some unit tests in CustomerDaoTest. Again, Spring Boot will help us a lot here with setting up and shutting down properly the testing environment via annotations such as @RunWith and @SpringBootTest. We only need to declare an autowired object CustomerDao.

Spring offers powerful means for database testing, for instance, @Transational and @Rollback. With these annotations, Spring will take care of database transactions as well as rollling back the testing databases to its initial state.

Unfortunately, it would be too tedious to annotate each test method. Spring Boot comes to handy with @JdbcTest that enables several useful features for JDBC tests including an in-memory embedded database as well as transactional and roll back at the end of each test.

IMPORTANTBy default, @JdbcTest won’t load regular Component, and as a result, CustomerDao won’t be loaded because it is a @Repository, i.e. a sub-type of @Component. Spring will inform us that it fails to find the required autowire CustomerDao for CustomerDaoTest. To fix this, simply annotation CustomerDaoTest with @ComponentScan.

This is the whole test case for our tentative CRUD methods.

package io.github.htr3n.springjdbcsimple.dao;

import io.github.htr3n.springjdbcsimple.entity.Customer;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jdbc.JdbcTest;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.test.context.junit4.SpringRunner;

import java.util.List;
import java.util.Optional;
import java.util.Random;

import static org.assertj.core.api.Assertions.assertThat;

@RunWith(SpringRunner.class)
@JdbcTest
@ComponentScan
public class CustomerDaoTest {

  private static final String ALICE_NAME = "Alice";
  private static final String ALICE_EMAIL = "alice@test.com";
  private static final String BOB_NAME = "Bob";
  private static final String BOB_EMAIL = "bob@test.com";
  private static final int ONE_CUSTOMER = 1;
  private static final int TWO_CUSTOMERS = 2;

  @Autowired
  private CustomerDao customerDao;

  private Customer alice;
  private Customer bob;

  @Before
  public void setUp(){
    alice = new Customer();
    alice.setName(ALICE_NAME);
    alice.setEmail(ALICE_EMAIL);

    bob = new Customer();
    bob.setName(BOB_NAME);
    bob.setEmail(BOB_EMAIL);
  }

  @Test
  public void create_shouldReturnValidCustomer_whenAddingNewCustomer() {
    customerDao.create(alice);
    assertThat(alice.getId()).isNotNull();

    Optional<Customer> result = customerDao.findById(alice.getId());
    assertThat(result).isPresent();
    assertThat(alice).hasFieldOrPropertyWithValue("name", ALICE_NAME);
    assertThat(alice).hasFieldOrPropertyWithValue("email", ALICE_EMAIL);
  }

  @Test
  public void findById_shouldReturnInvalidCustomer_forEmptyDatabase() {
    Optional<Customer> invalidCustomer = customerDao.findById(new Random().nextInt());
    assertThat(invalidCustomer.isPresent()).isFalse();
  }

  @Test
  public void findById_shouldReturnValidCustomer_forExistingCustomer() {
    customerDao.create(alice);
    Optional<Customer> validCustomer = customerDao.findById(alice.getId());
    assertThat(validCustomer).isPresent();
    assertThat(validCustomer.get().getName()).isEqualTo(alice.getName());
    assertThat(validCustomer.get().getEmail()).isEqualTo(alice.getEmail());
  }

  @Test
  public void findAll_shouldYieldEmptyList_forEmptyDatabase() {
    List<Customer> noCustomers = customerDao.findAll();
    assertThat(noCustomers).isNullOrEmpty();
  }

  @Test
  public void findAll_shouldYieldListOfCustomers_forNonemptyDatabase() {
    customerDao.create(alice);
    List<Customer> customers = customerDao.findAll();
    assertThat(customers).isNotNull().hasSize(ONE_CUSTOMER);

    Customer result = customers.get(0);
    assertThat(result).hasFieldOrPropertyWithValue("name", ALICE_NAME);
    assertThat(result).hasFieldOrPropertyWithValue("email", ALICE_EMAIL);

    customerDao.create(bob);
    customers = customerDao.findAll();
    assertThat(customers).isNotNull().hasSize(TWO_CUSTOMERS);
  }

  @Test
  public void update_shouldYieldFalse_forEmptyDatabase() {
    Customer notFound = new Customer();
    notFound.setId(new Random().nextInt());
    assertThat(customerDao.update(notFound)).isFalse();
  }

  @Test
  public void update_shouldYieldTrue_forExistingCustomer() {
    customerDao.create(alice);
    assertThat(alice.getId()).isNotNull();
    assertThat(customerDao.update(alice)).isTrue();

    alice.setName(BOB_NAME);
    alice.setEmail(BOB_EMAIL);
    assertThat(customerDao.update(alice)).isTrue();

    Optional<Customer> found = customerDao.findById(alice.getId());
    assertThat(found).isPresent();
    assertThat(found.get().getName()).isEqualTo(alice.getName());
    assertThat(found.get().getEmail()).isEqualTo(alice.getEmail());
  }

  @Test
  public void delete_shouldYieldFalse_forEmptyDatabaseOrNonexistingCustomer() {
    assertThat(customerDao.delete(new Random().nextInt())).isFalse();
  }

  @Test
  public void delete_shouldYieldTrue_forExistingCustomer() {
    customerDao.create(alice);
    assertThat(customerDao.findAll()).hasSize(ONE_CUSTOMER);
    assertThat(customerDao.delete(alice.getId())).isTrue();
    assertThat(customerDao.findById(alice.getId()).isPresent()).isFalse();
    assertThat(customerDao.findAll()).isEmpty();
  }
}

CRUD Implementation

With the tests defined, we can start implementing the CRUD using Spring JdbcTemplate.

Create

To satisfy the test described in create_shouldReturnValidCustomer_whenAddingNewCustomer(), our Customer.create() must successfully save the input Customer and return with an auto-generated primary key (e.g. customer ID). In order to obtain the key, we can use Spring’s helper class KeyHolder along with the standard PreparedStatement. Note that the code was simplified with Java 8 Lambda notation.

// Create
public Customer create(Customer customer) {
  String sql = "INSERT INTO customer (name, email) VALUES (?, ?)";
  KeyHolder keyHolder = new GeneratedKeyHolder();
  
  this.jdbcTemplate.update(connection -> {
    PreparedStatement statement = connection.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS);
    statement.setString(1, customer.getName());
    statement.setString(2, customer.getEmail());
    return statement;
  }, keyHolder);
  
  Integer newCustomerId = keyHolder.getKey().intValue();
  customer.setId(newCustomerId);
  return customer;
}

Please note that, Spring also provides another helper, namely, SimpleJdbcInsert with methods executeAndReturnKey(). Per the class’s documentation, SimpleJdbcInsert is indeed a higher level wrapper of JdbcTemplate.

Retrieve

In this project, we develop two retrieval methods: findAll() will return a list of all customers whilst findById() will look for a certain customer using the input ID.

The method findAll() is rather a piece of cake but findById() is quite tricky. Most of the tutorials or guides on Spring JdbcTemplate I have found on the Internet blindly use the method JdbcTemplate.queryForObject() to look up a database row. Trust me, I made the same mistake, too, along the line of this.

public Optional<Customer> findById(Integer id) {
  String sql = "SELECT id, name, email FROM customer WHERE id = ?";
  return Optional.of(jdbcTemplate.queryForObject(sql, new Object[] { id }, new CustomerMapper()));
}

Neater and cleaner, eh!? Unfortunately, the aforementioned findById() based on the problematic methods JdbcTemplate.queryForXXX() which is recommended to use in case the query shall return a single row and fails miserably in other cases.

This issue has been exposed by the test findById_shouldReturnInvalidCustomer_forEmptyDatabase() and I was able to come up with a slightly better version that uses JdbcTemplate.query() and checks for the returning ResultSet, in case of non-exiting customer yielding correctly null.

// Retrieve
public List<Customer> findAll() {
  String sql = "SELECT * FROM customer";
  return this.jdbcTemplate.query(sql, new CustomerMapper());
}
// Retrieve
public Optional<Customer> findCustomerById(Integer id) {
  String sql = "SELECT id, name, email FROM customer WHERE id = ?";
  return this.jdbcTemplate.query(sql,
    rs -> rs.next() ? Optional.of(new CustomerMapper().mapRow(rs, 1)): Optional.empty(),
    id);
}

When working with pure JDBC, we must map the database query result onto the domain entity on our own. This can be quickly done by implementing the interface RowMapper<T>.

class CustomerMapper implements RowMapper<Customer> {
  @Override
  public Customer mapRow(ResultSet rs, int rowNum) throws SQLException {
    Customer customer = new Customer();
    customer.setId(rs.getInt("id"));
    customer.setName(rs.getString("name"));
    customer.setEmail(rs.getString("email"));
    return customer;
  }
}

Update / Delete

The implementation of updating and deletion is rather straightforward.

// Update
public boolean update(Customer customer) {
  String sql = "UPDATE customer SET name=?, email=? WHERE id=?";
  Object[] params = new Object[]{customer.getName(), customer.getEmail(), customer.getId()};
  return this.jdbcTemplate.update(sql, params) == 1;
}

// Delete
public boolean delete(Integer id) {
  String sql = "DELETE FROM customer WHERE id = ?";
  Object[] params = new Object[]{id};
  return this.jdbcTemplate.update(sql, params) == 1;
}

Finally, we finish implementing all necessary CRUD methods and hopefully pass all the aforementioned tests.

You can check out the project source code available on Github.

Handy macOS QuickLook configurations

Mon, 30 Jul 2018 00:00:00 +0000

QuickLook is a simple but powerful feature of macOS that helps to preview the content of a file or folder in Finder without using any particular apps. This feature is very handy when we only want to have brief glance on a certain file or quickly examine its parts instead of opening a full-fledged app for that particular file.

Even nicer, Apple provides API for QuickLook programming and enables the use of plugins that convert a certain document from its native format into QuickLook previewing format. Before, it was also possible to also select and copy text from QuickLook previewing but then Apple removed that feature since macOS 10.11 El Capitan.

QuickLook is well supported in Spotlight search. In Finder, it can be conveniently activated by selecting a file or folder and then, for normal previewing, press Space or ⌘+Y. For full-screen previewing, you can press ⌥+Space (or using ⌘+⌥+Y in case you have ⌥+Space bound to some apps, for instance, Alfred in my case).

We might also use the command qlmanage but it works properly with built-in support and has issues with some external plugins.

# show the preview of file
qlmanage -p <path_to_file>

# or only show the thumpnail
qlmanage -t <path_to_file>

QuickLook Settings

QuickLook looks for and uses the plugins from both system-wide folders ‘/System/Library/QuickLook’ and ‘/Library/QuickLook’ and also the home folder of each user in ‘~/Library/QuickLook’.

A QuickLook plugin is organised in a folder with an extension .qlgenerator. The main task of a plugin, as mentioned above, is to generate previewing contents for the corresponding file types. We can see an example of the built-in plugin Image.qlgenerator.

$ tree /System/Library/QuickLook/Image.qlgenerator
/System/Library/QuickLook/Image.qlgenerator
└── Contents
    ├── Info.plist
    ├── MacOS
    │   └── Image
    ├── _CodeSignature
    │   └── CodeResources
    └── version.plist

The command qlmanage is provided in macOS since the announcement of QuickLook in 2007. It’s very helpful for manipulating QuickLook, for instance, reloading the generators, resetting cache, and many more. We will walk through some handy usage of qlmanage.

Forcing reloading the list of generators

qlmanage -r
qlmanage: resetting quicklookd

Showing the list of generators

This command can be used to see the current QuickLook generators list and also check which generator handles what file types. You might then see a long list of generators in your macOS.

qlmanage -m plugins

Resetting QuickLook caches

qlmanage -r cache

For more usage of qlmanage, we can look into its manual.

man qlmanage

QuickLook Plugins

Looking around, we can easily find a lot of plugins to enhance the built-in support of macOS. Moreover, when we install some applications, there might be included QuickLook plugins too.

Here is a list of plugins that I find very handy and use quite often in my macOS development box.

QLStephen

QLStephen is sort of an awesome Swiss-army-knife that I ever used. It enables the preview of numerous plain text files not handled by the default support of macOS QuickLook, especially files having no extensions such as README, CHANGELOG, Makefile, Rakefile, etc.

If you have already installed Homebrew, it takes only a simple command to install QLStephen (in case you haven’t, please feel free to head to another post).

brew cask install qlstephen

Figure 1. Using QLStephen to preview a Makefile (which has no extension and is not supported by macOS)

QLColorCode

I often quickly review my code using this amazing plugin forked/derived from the original one developed by Nathan Gray (n8gray). The preview code will be nicely formatted and highlighted.

brew cask install qlcolorcode

The default setting of QLColorCode is totall fine. Nonetheless, you might want to customise some of its settings to your tastes too. Here are some of my favourites.

defaults write org.n8gray.QLColorCode textEncoding UTF-16
defaults write org.n8gray.QLColorCode webkitTextEncoding UTF-16
defaults write org.n8gray.QLColorCode font "Source Code Pro"
defaults write org.n8gray.QLColorCode fontSizePoints 10
defaults write org.n8gray.QLColorCode hlTheme zenburn
defaults write org.n8gray.QLColorCode extraHLFlags "-W -J 160"
defaults write org.n8gray.QLColorCode pathHL /usr/local/bin/highlight

Here is the result of previewing Java code.

Figure 2. Code preview with syntax highlighting

QLCommonMark

I have been working with Markdown a lot, especially writing this blog all using Markdown and Hugo generator. It was quite a while I have searched around for a stable QuickLook plugin for Markdown. Before, I used MDD-QuickLook, an excellent fork of the well-known Fletcher’s original plugin. MDD-QuickLook allows me to customise the look of the Markdown preview using a Github based CSS file ~/.mdqlstyle.css.

When I found QLCommonMark, it replaces MDD-QuickLook with a broader coverage including CommonMark and Markdown and can be configured in the same way, just different filename ~/.cmqlstyle.css. You can find my Markdown preview style here.

brew cask install qlcommonmark

Figure 3. Markdown preview with a Github based style

QuickLook JSON

Both front-end and back-end developers are somehow familiar with JavaScript Object Notation (JSON) format. As JavaScript and its ecosystem are rapidly growing, JSON and its variations become extremely popular. Hence, a QuickLook plugin for JSON should be ready in your macOS dev box.

brew cask install quicklook-json

QuickLook CSV

CSV is a very popular format due to its simplicity and commonly used in statistics and data analytics. It contains values separated by comma, tabs, semi-colons, or pipes. MacOS provides built-in support for CSV previewing but I prefer a nicer-looking preview with QuickLook CSV developed by Pascal Pfiffner (p2) with alternate row shading.

brew cask install quicklook-csv

Figure 4. A CSV file preview

HetimaZip

There are many formats and standards around for compressing and archiving files such as 7z, bzip2, gzip, rar, zip, to name but a few. HetimaZip can help to look through a Zip archive without unzipping or opening it. Note that, this plugin also support well some file types based on Zip format such as Java ARchive (.jar), iOS app (.ipa), and many more.

brew cask install hetimazipql

Unfortunately, HetimaZip cannot handle Android application package (.apk). Thus we can combine with another nice plugin, QuickLook APK.

brew cask install quicklookapk

There is an alternative to HetimaZip, namely, BetterZip. I have tried both but could not make BetterZip’s work in macOS High Sierra like HetimaZip’s to show the content of a normal Zip file.

Figure 5. Looking through a Zip file content

Suspicious Package

MacOs applications are sometimes distributed as installer packages (.pkg). An installer package may run some scripts and install many files in your systems. A curious and careful user would need to find out what exactly the installation scripts doing and what files are going to be installed.

Advanced macOS users can use Show Package Contents to get inside the package, but it won’t show much details. We can use the app Suspicious Package or its included QuickLook plugin to inspect any installer packages before approving for installation.

brew cask install suspicious-package

Note that it is a tad inconvenient that you have to install the app Suspicious Package only for its plugin. Copying the plugin alone won’t work because it depends on the core libraries for previewing. I’m not quite sure whether this is the developers’ intention or there are actual technical difficulties behind the scene to solely deliver the QuickLook plugin.

Figure 6. Looking through an installer package

qlImageSize

Normally macOS provides sufficient plugins for quick-looking images. qlImageSize can enhance further by displaying the dimensions and size of the previewed image. These information is only available either when opening the image with an application or selecting the image file and pressing ⌘+I in Finder. I like qlImageSize a lot as many times I just want to quickly see an image’s dimensions and size this way.

brew cask install qlimagesize

Faster and enjoyable ZSH (maybe)

Mon, 23 Jul 2018 00:00:00 +0000

Z shell (Zsh) has been my shell of choice in both Linux and macOS. I used to install oh-my-zsh or the claimed-to-be-faster Prezto or Zim to leverage some of their cool libraries.

Unfortunately, I realised that my favourite terminal app, iTerm 2, became more and more sluggish when loading a new tab or window with more than one and a half second (without some virtual environment loaders like nvm, rvm, rbenv, jenv).

A quick G-fu could yield numerous posts on how to debug, optimise, speed up Zsh. I spent some time to refactor my own lightweight scripts instead of using big frameworks and was able to reach nearly one third of a second. In this post, I will share and discuss some aspects that might affect Zsh loading time and how to mitigate them. Most of the relevant code and configurations can be found here.

TL;DR:There are many aspects in Zsh that can potentially slow down its startup time and can be mitigated.

Organising shell startup order

Avoiding creating subprocesses

Lazy-loading instead of eager-loading

Optimising completion system

Optimising shell prompts

MacOS-specific optimisations

Performance Analysis

Overall execution time

Crunching some numbers on Zsh loading time would be definitely useful for further in depth analysis of its performance. A simple timing of Zsh startup time can be measured approximately using the time command that is available in most Unix/Linux/Mac systems.

$ /usr/bin/time /bin/zsh -i -c exit
        0.67 real         0.41 user         0.26 sys

The output of the command shows the execution time of Zsh breaking down to user-land and system. In order to get a better approximation, we can make a loop that invokes time for 10 or greater, if possible.

for i in $(seq 1 10); do /usr/bin/time /bin/zsh -i -c exit; done;

This timing method is very fast and handy in case you want to quickly see how your Zsh performs, especially to test some changes you have just made.

Profiling

Zsh provides a built-in module zsh/zprof that can be used to profile Zsh functions. At the beginning of ~/.zshrc, we add zmodload zsh/zprof. After restart the shell, we can use the command zprof to show a very rich output on Zsh startup loading. An illustrative output of zprof is shown below.

$ zprof
num  calls                time                       self            name
------------------------------------------------------------------------------
 1)    1          51.31    51.31   22.68%     49.68    49.68   21.96%  zle-line-init
 2)    2          45.72    22.86   20.21%     45.72    22.86   20.21%  compaudit
 3)  195          34.71     0.18   15.34%     25.52     0.13   11.28%  _zsh_autosuggest_bind_widget
 ...

Note

Two aforementioned approaches can give us a rough analysis on what aspects are invoked during Zsh startup so that we might figure out the bottleneck. In case you need to dig deeper, Arnout wrote a nice article in which he suggested a in-depth analysis and visualisation of Zsh loading using xtrace and kcachegrind. Benjamin developed a similar approach to more thorough tracing and analysis Zsh execution.

Problematic Aspects and Mitigation

Using two simple methods mentioned above, I was able to roughly understand some issues of my Zsh settings and tried to mitigate them to reduce startup time. I could not report the exact steps what have been done as it was a lot of trial-and-errors. Here I will discuss some major aspects combining my actual experiment and G-fu research.

Organising shell startup order

The order that Zsh loads its configuration files are documented here and here. Peter Ward drew a nice diagram showing the loading process of Zsh alongside with Bash and Sh (note that he omitted the system-wide configurations in Zsh part).

Figure 1. Bash, sh, and zsh shells startup order [source]

It’s also useful to understand relevant shell concepts, such as login vs. non-login, interactive vs. non-interactive shown in the diagram. Please refer to, for example, here or here for further explanations.

Some shell examples:

when opening a terminal emulator app (e.g. Terminal or iTerm 2), we are creating an interactive, non-login shell (please see login -pf later).

when logging in into a machine using command line ssh or su - username, we are working with an interactive login shell.

when executing a shell script, it is on a non-interactive, non-login shell.

Grokking Zsh startup order will help us putting relevant configurations in right files as well as optimising the loading process. My local Zsh setting is orgranised as follow:

.zshenv: invoked by all invocations of Zsh, so we should keep it small and merely initialise necessary variables.
.zlogin: will be loaded in login shells after .zshrc. My .zlogin will compile zcompdump in background as this is time-consuming and done only once per log-in session.
.zprofile : similar to .zlogin but will be sourced before .zshrc. Note that, both .zprofile and .zshrc are skipped in non-login non-interactive shells, as shown in Figure 1. So I learned a trick from Prezto that declares environment variables in .zprofile and uses .zshenv to source .zprofile (e.g. .zprofile and .zshenv). This way, non-login non-interactive shells will receive proper variable initialisations.
.zshrc: will be sourced in interactive shells. This contains the main part of Zsh configuration (e.g. my .zshrc).

Avoiding creating subprocesses

Some commands look totally innocent and standard in your shell scripts but might cost you dearly. Among them are commands that launch new child processes such as command substitutions and eval commands.

Command Substitutions

In Zsh, a command enclosed in $(command) or quoted with backticks `command` will be replaced with its standard output. Thus, it is very popular in Unix/Linux world when one wants to execute a command and get the output to process further on. The bad thing is that command substitution will launch a new process (i.e. a subshell).

eval command

eval [arg...]

The command eval (sounds with evil, ^_O) is part of POSIX standard and is available in most shells. It’s often used together with command substitution. Essentially, eval concats its arguments separated with spaces along with evaluating any variables or expressions to form a command with or without arguments. Then it executes the resulting command in the current shell. As such, eval will cause side-effects as it must evaluate the inputs and perform expansions, if any.

$ command="print 'Hello World'"
$ command			   # nothing happen
$ eval $command
Hello World

This feature makes eval powerful as it can dynamically evaluate and execute code. But dynamic evaluation also makes eval risky and time-consuming.

In some cases, for instance, simple string manipulation, we can consider to replace command substitutions and eval commands that invoke sed, awk, etc., with Zsh built-in constructs or hard-coded constants. Zsh provides numerous powerful built-in mechanisms for substring matchings, string explosion/splitting, and expansions.

For instance, when using Homebrew, it is very convenient to get the path to an installed package using $(brew --prefix <package>).

export PATH=$PATH:"$(brew --prefix httpd)/bin"

It turns out many calls to $(brew --prefix ...) would launch many subprocesses and thus slow down Zsh notably. When replacing that $() command with its actual output, I could gain certain improvement. The caveat is that some upgraded versions might break these hard-coded values.

Lazy-loading instead of eager-loading

Function autoloading

We can define and source new functions in Zsh. In this way, a function is eagerly loaded and always available for use. Note that most of these functions might be not really needed until being invoked.

Zsh can help postponing their loading time and allow to load-on-demand via function autoloading. This technique is often called lazy loading. Performance-wise, lazy loading will put less pressure to the underlying system and reduce memory footprint. The same techniques are also preferred in many other fields such as databases, dynamic runtime libraries, etc.

In my codebase, I create a folder, namely, autoloaded, to store functions that will be, er…, autoloaded by Zsh. For each function, for instance, function hello(){...}, I will create a corresponding file named hello inside autoloaded. The content of that file is the function body (i.e. without function and (){}).

$ mkdir autoloaded
$ echo "print 'Hello World'" >> autoloaded/hello
$ tree
.
└── autoloaded
    └── hello

The folder autoloaded must be added to ZSH variable fpath where ZSH will look for function definitions.

# add 'autoload' to fpath
$ fpath=($fpath autoloaded)
# try to invoke 'hello'
$ hello
zsh: command not found: hello
# now mark `hello` for autoloading.
$ autoload hello
# quickly check how `hello` will be loaded.
$ which hello
hello () {
	# undefined
	builtin autoload -X
}
# now it works fine
$ hello
Hello World

The body of hello was marked with #undefined along with builtin autoload -X meaning it will be loaded on-demand. The first time hello is called, Zsh will automatically load and execute it.

In the same way, I configured Zsh to load all of my functions on-demand to reduce memory and loading time.

Note:In my scripts, autoload -Uz function_name was used. The option -U prevents alias from being expanded. That is, whenever you define an alias and a function having the same name, the alias will be considered first instead, so -U just skips alias expansion. And the option -z indicates that the function will be auto-loaded using zsh or ksh style.

Selective- or lazy-loading virtual environments

Many virtual environment loaders like rvm, rbenv, jenv, nvm have been developed to manage different run-time versions and libraries. While being very handy for software development, most of these tools need to be eagerly loaded (e.g. directly source in .zshrc) to work properly.

Instead, we can consider to transform these loaders as much as possible into on-demand wrapper functions. You can find a good example here. In summary, Peter’s trick is to override nvm with his own autoloaded nvm() (that eventually invokes the original nvm loader). Carlos also went to same way for rbenv, his own antibody, pyenv and achieved some good results. Benny C. Wong did similarly for both nvm and rvm.

You can also find another interesting post by Frederic about optimising Zsh loading time by converting Kubernetes’s initialisation code into a lazy-loading function. When not using lazy-loading, you might find Adam’s trick useful for reducing rbenv time.

Optimising completion system

One of the beloved Zsh’s features is its new completion system, so-called zshcompsys. That is, when you type half of a certain command and press Tab, Zsh is able to show some suggestions for completing that command.

Zsh does ship with some built-in support for popular commands but not for all kinds of commands. Instead, Zsh offers powerful means for defining custom completion via underscored-autoloaded files.

Zsh completion system must be activated by calling function compinit. Most of the framework like oh-my-zsh or Prezto will take care of initialising completion system. In my case, after getting rid of big frameworks, I have to manually activate it with autoload -Uz compinit && compinit.

Every time compinit is invoked, it often checks its configurations and re-generates in case of changes. Some have investigated this matter and suggested improvements such as checking compinit’s cache only once a day. A similar approach has also been implemented in Prezto.

Here is a simple excerpt based on Carlos’s solution.

autoload -Uz compinit
if [ $(date +'%j') != $(/usr/bin/stat -f '%Sm' -t '%j' ${ZDOTDIR:-$HOME}/.zcompdump) ]; then
  compinit
else
  compinit -C
fi

Lessening `compinit` invocations

When running zprof to profile Zsh execution, as many others also found out, I noticed a lot of invocations to compinit. It was because I used some smart plugins like zsh-users/zsh-completions and zsh-users/zsh-autosuggestions and scattered compinit in many places. Using ack, I could quickly spot and remove all compinit, then only call once at the end of my .zshrc.

Compiling completion dumped files

Note that by default compinit will produce a dumped configuration for accelerating future access. The default dumped file is .zcompdump (which can be changed with compinit -d new_dump_file or totally disabled with compinit -D).

We can go further by compiling the dumped file with the built-in command zcompile for faster autoloading of completion functions. As completion is only needed for interactive shell sessions, I put the zcompile code inside .zlogin and force it to run in background mode.

# Execute code in the background to not affect the current session
{
  # Compile zcompdump, if modified, to increase startup speed.
  zcompdump="${ZDOTDIR:-$HOME}/.zcompdump"
  if [[ -s "$zcompdump" && (! -s "${zcompdump}.zwc" || "$zcompdump" -nt "${zcompdump}.zwc") ]]; then
    zcompile "$zcompdump"
  fi
} &!

Optimising shell prompts

You can find a lot of frameworks or plugins offer super duper cool command line prompts like this, this, or these that show rich information regarding your current working folder such as versioning status, virtual environments, and many more.

Obviously getting these information will induce extra execution time, especially for checking large versioned repositories or virtual runtime libraries. That leads to many workarounds, tweaks, hacks, other hacks, and more hacks.

You might want to consider some recent approaches on speeding up shell prompts such as Anish’s non-blocking prompt or Sindre Sorhus’s pure based on Mathias Fredriksson’s zsh-async. I have tried pure and found out its timing is very close to my own prompts based on vanilla Zsh scripts and built-in function vcs_info. Therefore, I mostly switch back and forth between these prompts in my dev box and totally satisfy with their performance thus far.

MacOS-specific optimisations

Optimising `path_helper`

In the chain of Zsh startup order, /etc/zprofile will be sourced before ~/.zprofile. So, macOS uses /etc/profile to establish paths to executable files via path_helper.

$ cat /etc/zprofile
# system-wide environment settings for zsh(1)
if [ -x /usr/libexec/path_helper ]; then
	eval `/usr/libexec/path_helper -s`
fi

To do that, path_helper will read /etc/paths and /etc/manpaths, then read all files inside /etc/paths.d and /etc/manpaths.d and append their contents to $PATH and $MANPATH, respectively. New paths can be conveniently added by adding a plaintext file in /etc/paths.d instead of messing common shared configuration files.

Previously in some cases, path_helper might be very slow as mentioned by Michael Tsai here and even deserved a patch and a Perl based alternative. I reckoned that path_helper is getting notably slow when the number of paths are growing but its recent version is no longer a script but 64-bit binary executable and seems to work faster.

If you notice that path_helper makes Zsh slow, you can just put the contents of /etc/paths and of all files in /etc/paths.d directly in .zprofile . After that, just comment out the corresponding code in /etc/profile.

The default login process of macOS could be the culprit too. Opening a terminal window or tab will trigger login -pf username which, in turn, reads from and writes into the logs file in /var/log/asl (note syslog() invocations in login.c).

We can check this out in any terminal app. The command ps -ef | grep login will show details about the login process.

$ ps -ef | grep login
...
  0 25142 25141   0  6:40AM ttys000    0:00.13 login -fp htr3n

In case you want to dig deeper, execute the following command in a terminal tab/window,

$ sudo opensnoop | grep "/var/log/asl"

then open another tab or window to see a lot of accessing to ASLs (standing for Apple System Log, deprecated since macOS 10.12 but still in use somewhere).

Thus, some have reported a magic that somehow speeds up shell startup by deleting macOS ASLs.

# remove all Apple system logs -- more destructive way
$ sudo rm -rf /private/var/log/asl/*.asl

Nevertheless, these logs will continuously grow day after day. We might configure /etc/asl.conf to permanently reduce the amount of ASLs. Using sudo to open that file in Text Editor (or your editor of choice),

$ sudo open -e /etc/asl.conf

then looking for the following lines and changing them accordingly and leaving the rest intact.

...
# save everything from emergency to notice
# ? [<= Level notice] store
? [<= Level critical] store
...

The idea is to change the log level from notice to a higher level such as warning, error, or critical (see more on Syslog Message Severities in RFC 5424. You might also look further in the folder /etc/asl/ to tinker log configurations of certain applications but that is beyond the scope of this article.

Another way is to skip the process of accessing ASLs altogether. For instance, with iTerm 2, press ⌘ + , to open menu Preferences , then go to Profiles ▸ General .

Figure 2. iTerm 2 login setting

The same way can be applied for Apple’s built-in Terminal app.

Figure 3. Built-in Terminal login setting

Conclusion

We have walked through some major aspects that might affect Zsh in particular, and other shells, loading time. I hope these discussions can help you to pinpoint and address your shell startup issues and have better experience working with shells and command line. This is what I got after all these effort.

❯ for i in $(seq 1 5); do /usr/bin/time /bin/zsh -i -c exit; done
        0.31 real         0.16 user         0.13 sys
        0.28 real         0.15 user         0.12 sys
        0.28 real         0.15 user         0.12 sys
        0.28 real         0.15 user         0.12 sys
        0.28 real         0.15 user         0.12 sys

If you have any suggestions for improvement or successful tweaks, please drop a comment below.

Reading List

A mininal Hugo blogging workflow

Sat, 14 Jul 2018 00:00:00 +0000

Hugo is a blazingly fast static site generator based on Go. My first experience with Hugo was truly pleasant. I appreciate the separation of contents, presentations, and site generation logics in Hugo that leads to less effort for maintenance or switching themes.

On the run-time side, it’s also nice that Hugo is often delivered as a reasonably fat executable binary (about 30~50 MB). We almost do not have to install anything else to make it work (although this is not entirely true since Hugo 0.43+).

Using Hugo to build this blog, I have learned a lot of things and tried to set up a simple blogging workflow to automate as much as possible the process from writing articles and altering themes to generating sites and watching for changes. My Hugo based blogging workflow comprises three aspects:

Content (mainly Markdown)
Presentation (theming)
Site generation (Hugo)

I will take my own Hugo blogging project as an illustrative example. An overview of the project structure is shown in the picture below.

Figure 1. A simple Hugo project structure

Some relevant project resources:

assets: my pre-processing resources (e.g. SASS, LESS, images, JavaScript).
content: Markdown contents
public: the Web sites generated by Hugo
static: resources (e.g. CSS, images, JavaScript) to be copied directly to public by Hugo.
themes: Hugo themes (here I use my own theme hyde-hyde)
package.json: Node.js / NPM main configuration
gulpfile.js: Gulp configuration

Content

Among many nice features, Hugo supports Markdown contents out-of-the-box. Thus I can keep writing blog posts with my favourite Markdown editors. I have tried a number of editors in macOS, ranging from simple plain text with preview add-ons like Sublime Text, TextMate, Atom, Visual Studio Code, Vim to some built-in ones like Bear, Byword, MacDown, Mou, Texts, IA Writer, Marked (only preview), Typora, to name but by no means completed. You can see here a list including most of Markdown editors available and here are some for macOS.

My first editor was Mou, due to its simplicity and highly customisable themes and rendering styles. Later on Mou’s developer decided to sell its ownership, then resumed working on its beta making the software status a bit unclear and frustrated. Tzu-ping Chung (@uranusjr) has created a similar editor, namely, MacDown, and open-sourced it. Then I started using MacDown with the existing Mou’s themes and styles.

I could not remember exactly when and how I found Typora but it never ceases to amaze me. With Typora, I can enjoy seamless editing and rendering in the same window, i.e., I can write Markdown normally and directly see the content rendered. Typora is still free in its beta phase and will become paid software when released. It has become my Markdown editor of choice and will be in future should its price tag is reasonable. For quick fixes, I use any editors at hand, for instance Visual Studio Code when working on a Hugo project.

Presentation

The nice thing is that a Hugo theme is a standalone component for decorating Hugo generated sites. Nevertheless, I can override a theme’s settings (e.g. layouts, styles) with my own versions with respect to Hugo lookup order. A theme contains resources for templating, layouting, and styling. Hugo’s templating and layouting have been discussed a lot around. Styling resources in a typical Hugo project can be found inside its theme’s static or the project root’s static and assets.

I used to work with CSS styles. After learning more about CSS preprocessors and attracted by the abstraction levels and tooling they provide, I decide to give a try and refactored my existing codebase to SCSS. Hence, my workflow should include support for SCSS (you might opt for LESS or Stylus but still can leverage a similar workflow with just right processing components).

Site Generation

Hugo will be main element that generates the final Web site taking input resources from content and presentation components. Until version 0.43, Hugo has not provided support for SASS. Hence, many have to develop their own ’asset pipeline’, i.e., incorporating preprocessors and post-processors with Hugo generation process.

Endogenous Pipeline

This approach, also called in-line pipeline, only works since Hugo v0.43/extended and later that supports processing SASS/CSS with consideration for minification, post-processing, and cache busting.

In Hugo templates, I can refer to SCSS files via page resources.

{{ $styles := resources.Get "scss/hyde-hyde.scss" | toCSS | postCSS (dict "use" "autoprefixer") | minify | fingerprint }}
<link rel="stylesheet" href="{{ $styles.Permalink }}" integrity="{{ $styles.Data.Integrity }}">

Note that Hugo will look for scss/hyde-hyde.scss in the project’s root and then the theme’s root, in that order.

Hugo retrieves the resource hyde-hyde.scss using Resources.Get, then transforms it into hyde-hyde.css and passes it to PostCSS for autoprefixing. After that, it minifies the CSS and generates a fingerprinted version. The second line simply links the resulting hyde-hyde.css with HTML via the variable $style. The result in the generated HTML will be like this (note that the code can be different to yours).

<link rel="stylesheet" href="http://localhost:1313/scss/hyde-hyde.4ff234ab46aa5302c7e0d2f35b9c76a8bba9fe42a9e8a6c7c47df7f85b8de122.css" integrity="sha256-T/I0q0aqUwLH4NLzW5x2qLup/kKp6KbHxH33&#43;FuN4SI=">

The fingerprint code is for cache busting and only changed when the source file is altered.

The biggest advantage of in-line pipeline is that Hugo already provides a certain level of built-in support. Hugo’s live reload, i.e. serving the resulting sites live and automatically refreshing whenever there are changes of contents or themes, also works superbly for this case.

For many scenarios, we can use this approach instead of configuring complex tools like Gulp, Grunt, or Webpack. To be precise, I do need to install extra tools, which are postcss-cli and autoprefixer, for the aforementioned example to work and Hugo 0.43 extended version partially depends on Cgo/libc++.

$ npm i -g postcss-cli autoprefixer

Another downside to consider is that, in this way, external tools like postcss will slow down Hugo.

Note

As I checked out this thread and tried on my project, Hugo’s extensions like postCSS, minify, and fingerprint could mess up CSS sourcemaps. If you need sourcemaps for developing and debugging, please ignore them and consider the exogenous pipeline.

Exogenous Pipeline

When working on more complex asset pipelining, we have to involve external tools to build an exogenous asset pipeline. This is also my blogging workflow before Hugo v0.43. Now, I can mostly get rid of this exogenous pipeline. Here we will walk through this approach merely for the sake of completeness.

There are several ways to handle Web resources. For instance, Netlify provides a boilerplate project, namely, victor-hugo for starting a new Hugo project and preparing to deploy to its hosting. This boilerplate employs Gulp, Webpack, PostCSS, and Browsersync around Hugo. There are many developers rooting for Gulp [1, 2, 3, 4, 5] whilst some others go for Grunt [6] or Webpack [7, 8]. And even few brave ones use pure NPM [9, 10].

For starter projects with Hugo, you might want to consider some useful projects such as hugulp, Huggle, hugo-webpack-boilerplate. In my project, I combine Gulp with NPM and leverage some Gulp’s components for the asset pipeline.

SASS/SCSS to CSS: a sensible choice is gulp-sass.
Sourcemaps (optional): this is not mandatory but very handy for developing and debugging phase. I pick gulp-sourcemaps.
Autoprefixing: it helps adding vendor-specific prefixes to CSS rules. We can use gulp-autoprefixer but be careful to combine with gulp-sourcemaps as it is currently broken. I decide to use gulp-postcss along with postcss-autoprefixer instead.
Minification: PostCSS provides many plugins for CSS post-processing alongside autoprefixer. I choose cssnano to minify the CSSs.

$ npm i -D gulp@next gulp-sass gulp-postcss autoprefixer gulp-sourcemaps cssnano

In the same way, you can add more components to the list. Here I rather keep it short and simple for illustrative purpose. Now I start configuring Gulp to use them in my project.

const gulp = require('gulp');
const sass = require('gulp-sass');
const postcss = require('gulp-postcss');
const autoprefixer = require('autoprefixer');
const cssnano = require('cssnano');
const sourcemaps = require('gulp-sourcemaps');
const log = require('fancy-log');

const sassSourceFile = 'assets/scss/hyde-hyde.scss';
const outputFolder = 'static/css';
const watchedResources = 'assets/scss/**/*';

gulp.task('scss', function (done) {
  gulp.src(sassSourceFile)
    .pipe(sourcemaps.init())
    .pipe(sass().on('error', function(err){
      log.error(err.message);
    }))
    .pipe(postcss([autoprefixer, cssnano]))
    .pipe(sourcemaps.write('.'))
    .pipe(gulp.dest(outputFolder))
    .on('end', done);
});

gulp.task('watch', gulp.series('scss', function (done) {
  gulp.watch(watchedResources, gulp.parallel('scss'));
  done();
}));

gulp.task('default', gulp.series('watch', function () {}));

Note that the syntax is Gulp v4.0.

Now I just simply invoke Gulp to trigger its chaining pipeline.

$ gulp
[10:23:40] Using gulpfile ~/working/dev/htr3n-blog/gulpfile.js
[10:23:40] Starting 'default'...
[10:23:40] Starting 'watch'...
[10:23:40] Starting 'scss'...
[10:23:40] Finished 'scss' after 58 ms
[10:23:40] Starting '<anonymous>'...

Gulp will transform hyde-hyde.scss into hyde-hyde.css which is autoprefixed and saved in static/css along with its sourcemaps. In the Hugo templates, I refer to hyde-hyde.css normally.

<link rel="stylesheet" href="{{ .Site.BaseURL }}/css/hyde-hyde.css">

I can combine Gulp.watch() with Hugo’s live reload feature so that both will continuously monitors for changes and serve them instantly. This way, whenever I made any changes in a post in Typora, a SCSS/CSS file, or Go+HTML layout, I will instantly see the generated HTMLs.

$ hugo server -w --cleanDestinationDir
                   | EN
+------------------+-----+
  Pages            | 320
  Paginator pages  |   0
  Non-page files   |  14
  Static files     |  14
  Processed images |   0
  Aliases          |   0
  Sitemaps         |   1
  Cleaned          |   0

Total in 193 ms
Watching for changes in /Users/htr3n/working/dev/htr3n-blog/{assets,content,data,layouts,static,themes}
Watching for config changes in /Users/htr3n/working/dev/htr3n-blog/config.toml
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop

Please note that, by default, Hugo will only keep watching the following folders {assets, content, data, layouts, static, themes}. Hence, our SCSS files might not be in one of these folders but we must ensure the resources generated from Gulp are. As I illustrated endogenous pipeline above, my SCSSs are kept in assets such that Hugo Resources.Get can find them. If you opt for exogenous pipeline, you should put the preprocessing resources elsewhere.

After all, we can open the browser to http://localhost:1313 and enjoy the experience of live editing side-by-side.

Figure 2. Typora (left) and the generated HTML pages (right)

At this point, we seem to miss cache busting which is provided in the endogenous pipeline with Hugo. That is also the biggest advantage of using external tools as we need to establish the connections between Hugo generated resources and Gulp’s manipulated resources on our own.

It is a tad complicated as you can find here, here and here. In summary, the idea is to generate hash signatures for your CSSs, for instance, using MD5 or SHA, store the relevant data as dictionary key-value pairs, e.g. 'hyde-hyde.css' => 'hyde-hyde.the-latest-hash-number.css', then use Hugo’s data templates feature to refer to those pairs. This approach has actually been implemented in the aforementioned Hugo’s extensions.

In case you really need cache busting, we can utilise Hugo v0.43 extensions to complement Gulp on this matter (🐵). We just modify the Gulp configuration a bit so that the output folder is no longer static but assets/css as Hugo’s Resources.Get rather strictly looks for resources in assets.

var outputFolder = 'assets/css';

And in the HTML, we use similar templates without toCSS and postCSS that are done with Gulp.

{{ $styles := resources.Get "css/hyde-hyde.css" | minify | fingerprint }}
<link rel="stylesheet" href="{{ $styles.Permalink }}" integrity="{{ $styles.Data.Integrity }}">

It seems Gulp becomes redundant here as Hugo can perform toCSS, postCSS, and minify just fine. Nonetheless, in case you need other functionalities that are not yet provided by Hugo, Gulp will be what you should ask for. Besides, as per Note above, there are still some issues with Hugo’s extension regarding sourcemaps whilst Gulp’s asset pipeline above works flawlessly.

More automation, please!!!

There are still two manual tasks for the exogenous pipeline, one for invoking Gulp and another one to start Hugo. Aiming to automate the workflow as much as possible, we can merge two tasks using one execution script, via NPM.

There are a number of options for running tasks concurrently in NPM including npm-run-all, npm-run-parallel, concurrently, or using background process if you are using Linux/macOS alike. Even Gulp also has some helper plugins like gulp-exec to execute child processes, gulp-nodemon to run in daemon mode, and Gulp 4.0+ has Gulp.parallel().

Let’s consider a simple example with npm-run-all. Other tools can be used similarly.

$ npm i -D npm-run-all

Then define some simple scripts in package.json for executing Gulp and Hugo.

{
  "scripts": {
    "scss-build": "gulp scss",
    "scss-watch": "gulp watch",
    "hugo-watch": "hugo server -w --buildDrafts --cleanDestinationDir",
    "dev": "npm-run-all --parallel scss-watch hugo-watch"
  },
  "devDependencies": {
    "autoprefixer": "^9.0.0",
    "cssnano": "^4.0.3",
    "gulp": "^4.0.0",
    "gulp-postcss": "^7.0.1",
    "gulp-sass": "^4.0.1",
    "gulp-sourcemaps": "^2.6.4",
    "npm-run-all": "^4.1.3"
  }
}

Given no issues, we should execute npm run dev and see both Hugo and Gulp running in parallel.

Further Optimisations

As you might have noticed, the Web resources (e.g. HTML, CSS) generated by Hugo may contain a lot of whitespace and/or line-breaks. You can easily see this when viewing the source code of the generated Web pages.

During the developing phase in my local machines, I always keep the generated resources untouched so that I can read the code for debugging or analysing. When deploying to hosting servers, we don’t often preserve the readability of the source code as the end-users mainly see and interact with the beautiful pages rendered by Web browsers.

Therefore, I configure a simple Gulp task to optimise the generated Web resources before pushing them to the servers. What I need is gulp-htmlmin—a Gulp front-end of html-minifier. Please refer to html-minifier’s documentations should you need to learn further about its options.

$ npm i -D gulp-htmlmin

And I create a simple Gulp task for minifying Web resources that can be invoked at command line as gulp minify-html. This task should be performed after calling Hugo to generate the Web pages.

const htmlmin = require("gulp-htmlmin");
//...
gulp.task("minify-html", function(done) {
  let publicFolder = "./public";
  let html = publicFolder + "/**/*.html";
  let css = publicFolder + "/**/*.css";
  let js = publicFolder + "/**/*.js";
  let dest = "./dist";
  log("Minifying HTML/CSS/JS in '" + publicFolder + "' to '" + dest + "'");
  gulp.src(publicFolder + "/**")
    .pipe(gulp.dest(dest));
  gulp.src([html, css, js])
    .pipe(htmlmin({ collapseWhitespace: true }))
    .pipe(gulp.dest(dest))
    .on("end", done);
});

The main idea is to use gulp-htmlmin to minify HTMLs, JavaScripts, and CSSs whilst leaving others intact. Note that the place of the optimised resources is the folder dist inside Hugo project.

References

Bootstrapping macOS workspace

Wed, 27 Jun 2018 00:00:00 +0000

I have experienced a number of different Mac systems, ranging from the old white MacBook 2006 to iMac and MacMini 2010 and lately MacBook Pro 13" and 15". From time to time, Time Machine has become extremely handy for transferring my data from the old to the new machine.

Nevertheless, a lot of changes, mostly for software development, have often been ignored or reset to the default values. As a result, I often find myself searching around over and over for some desirable settings. Thus, this post is kind of a note-to-self on few useful tweaks for my development box so that I can easily reach out later.

Update: As it turned out that the tweaks and configurations were getting longer and became tedious to follow, I split them into two parts:

Part 1: System tweaks and configurations

Disable System Integrity Protection (SIP)

System Integration Protection (SIP) is a security technology since OS X El Capitan 10.11 aiming at protecting certain system files and folders against malicious software. SIP even limits the access and actions of the root user account on protected parts of OS X. Unfortunately, the folder /usr/local needed for installing many development tools and utilities is under protection as well. That makes a lot of tools like Homebrew fail to install. We can easily check SIP’s current status.

$ csrutil status
System Integrity Protection status: enabled.

The tweak I always do right after installing or recovering a Mac is to disable SIP. Just (re)start macOS in Recovery Mode (restarting or turning on your Mac and immediately pressing ⌘ + R—shortcut to remember R for ‘__R__ecovery’—and wait until macOS boots into its recovery mode. Then open menu Utility ▸ Terminal and execute the following commands.

$ csrutil disable
$ reboot

Your Mac will reboot and get back to the normal working mode. You can verify SIP status again.

$ csrutil status
System Integrity Protection status: disabled.

Disable Gatekeeper

Gatekeeper was first introduced in OS X Mountain Lion. It is a security feature that prevents the users from launching potentially harmful apps.

From macOS Sierra and later, however, Apple made Gatekeeper even more restricted. Normal users would see only two choices in System Preferences –> Security & Privacy that “Allow apps downloaded from” either “App Store” or “App Store and identified developers”.

That means only signed apps are allowed to launch, i.e. breaking a lot of useful unsigned apps whose developers cannot afford to pay US$99 yearly membership for Apple Developer Program.

Figure 1. Enabled Gatekeeper

We can circumvent that by disabling Gatekeeper. You do not have to restart macOS but only executing the following command sudo spctl --master-disable.

After that, you will be able to see Gatekeeper being disabled.

Figure 2. Disabled Gatekeeper

Naturally Gatekeeper can be re-enabled at any time with sudo spctl --master-enable.

Revealing ‘~/Library’

In macOS, the folder ‘~/Library’ within users home folder is hidden (since 10.7!?) seemingly to prevent any unintentional changes. It’s not so difficult to navigate to this folder (and any other folders) using command line. In case you want to use Finder to navigate and explore ‘~/Library’, we can simply use the following command.

$ chflags nohidden ~/Library

Enabling Key Repeating

The behaviour for keyboard long pressing in macOS is also of polarised personal taste. A typical behaviour that we often see is that the letter keeps popping out since one holds down a key until the key is released. Some years back this has been changed into ‘Character Picker’. It means when a key is pressed and held, the non-standard characters corresponding characters will pop up as the picture below.

Figure 3. Character Picker in action

Not a big fan of Character Picker, I rather like characters keep repeating when a key is held. In many cases, this behavior becomes very handy, for instance, when selecting large texts with ⇧ + ↓ ↑ ← →.

For that reason, I always disable that feature in any macOS that I am working on. Unfortunately, this cannot be done with System Preferences but command line. After each command, you must re-start the editors to see the effect.

# disable 'Character Picker'
$ defaults write -g ApplePressAndHoldEnabled -bool false
# it can be enabled
$ defaults write -g ApplePressAndHoldEnabled -bool true

Another option relating to key repeating is tuning its speed/rate and delay. Luckily, we can do this via System Preferences ▸ Keyboard . My favourite is ‘fastest’ repeating and ‘shortest’ delay as possible.

Figure 4. Setting for key repeating and delay

Like many other options, we can change it via command line or fine tuning tool such as Karabiner.

# InitialKeyRepeat: 120, 94, 68, 35, 25, 15 (lower is faster, default is 25)
$ defaults write -g InitialKeyRepeat -int 15
# KeyRepeat: 120, 90, 60, 30, 12, 6, 2 (lower is faster, default is 6)
$ defaults write -g KeyRepeat -int 2

Changing Hostnames

Most of the time, the default host name is not satisfied and you might want to have a cool name for your dev box. Then you are surely not alone. Here and there, you might find some cool combo like harry@hogwart , alice@wonderland, yoda@dagobah, joker@gotham, neo@matrix, frodo@mordor you name it. You can quickly do this via command line.

# change the primary host name
$ sudo scutil --set HostName gotham
# ... do the same for local (Bonjour) host name 
$ sudo scutil --set LocalHostName gotham
# and finally, keep the user-friendly computer's name in Finder in sync
$ sudo scutil --set ComputerName gotham

Part 2: Development tools and configurations

In this part, I will focus more on some handy tools and configurations often used for my software development tasks.

Homebrew

Homebrew provides a rich collection of tools I need, mostly open sources, that Apple does not. Installing Homebrew is super easy with a one-liner command. Before that, we only have to install the Xcode command line tools that Homebrew needs to build its tools.

# install Xcode command line developer tools
$ xcode-select --install

# now install Homebrew
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Then we need to add /usr/local/bin and /usr/local/sbin (where Homebrew links the executable files) to the environment variable PATH. This can be done by altering the shell startup scripts. For instance, for Bash shell it is ~/.bash_profile, for Zshell it is ~/.zsh_profile.

export PATH=$PATH:/usr/local/bin:/usr/local/sbin

If you want the Homebrew tools have higher priority than the others, just switch $PATH to the end instead. Moreover, there are many Finder-launched apps, agents and daemons in macOS might also need PATH to find necessary executables (e.g. Intellij IDEA and Eclipse if I’m not mistaken). Fortunately, the launching of these tools would go through macOS launchd and therefore, can be configure with launchctl. So I have to add this line to the end of my .zshrc too.

# this line should be added at the end of .zshrc
/bin/launchctl setenv PATH $PATH

Now I can install necessary tools using brew.

$ brew install ack curl dnsmasq docker gcc git go gradle maven mysql node openssl openssh python3 php72 composer ruby rbenv sassc yarn zsh

Homebrew can also help handling macOS start-up services which are useful for managing servers like Apache httpd, nginx, php-fpm, MySQL, PostgreSQL, etc.

$ brew services start httpd
$ brew services stop httpd

For quick re-installing or recovering Homebrew, I use homebrew-bundle for backing up the current Homebrew’s installation and restoring later.

# tap the bundle
$ brew tap Homebrew/bundle
# dump all existing Homebrew packages to Brewfile
$ brew bundle dump --force
# check the list of entries
$ brew bundle list
# restore/install all packages from the Brewfile 
$ brew bundle

Homebrew also incorporates Homebrew-Cask (formerly an independent plugin for Homebrew) that can help to install many macOS apps as you might see in the later sections.

Terminals and Shells

Since working with Linux a lot and then macOS, I’m rather familiar with the command line. There are many cases using terminals would be more convenient than graphical editors. Thus, I need a good terminal app. The built-in Terminal app in macOS is not bad but I usually opt for iTerm2.

$ brew cask install iterm2

Some of my favorite settings for iTerms are including dark themes such as Dracula, IR_Black, seoul_256, Atom One Dark along with a fixed-width font like Source Code Pro, Cousine, Roboto Mono, Fira Code or Operator Mono (very nice ligatures and italic mode), Inconsolata or its variants (Inconsolata-dz, Inconsolata-g), or Hack. Some of these fonts can be installed via Homebrew Cask.

Along with iTerms 2, I also love Z shell (zsh) which is an extended version of Bash shell a lot with many nice features like directory alias, command completion, path expansion, etc. You might also find some Zsh configuration frameworks like oh-my-zsh or Prezto amazing to start with. I’ve tried both of them and found oh-my-zsh a bit sluggish and Prezto a bit difficult to add more extensions or customize. I have actually used a small piece from either of them and found myself quickly forgetting the customised parts taken from the others. Hence, I decided to develop a set of my own Zsh configurations. Moreover, I often keep a habit of writting Shell scripts in standard sh/bash as much as possible and keep Zsh-specific separately for better portability and compatibility.

Text/Code Editors

There are quite a number of editors in macOS for developers ranging from, er, the built-in Text Editor, Sublime Text, TextMate, TextWrangler, the nerdy Vim, Emacs, and recently Atom, Visual Studio Code, Brackets, to name but a few. You might install these editos using Cask.

In macOS, Sublime Text is often a powerful editor for many of us but it is quite costly. I used to love and work with Atom developed by Github as an effort to build a modern editor for developers. Atom has a plethora of extensions to become a very promising choice. Nevertheless, I was growing skeptical with its sluggishness in early days and therefore leaned to Visual Studio Code, my current Swiss-army-knife editor-of-choice instead.

$ brew cask install visual-studio-code

Frequently using terminal/console, I just grasp vim for quick editing tasks and customised it a little bit.

$ brew install vim

As working with larger and complex projects, I would need some full-fledged, heavy-weight IDEs such as Eclipse or Jetbrains Intellij IDEA and its brothers like PhpStorm, PyCharm. Other IDE including Netbeans, Visual Studio, Komodo can also be installed with Cask.

$ brew cask install eclipse-java intellij-idea

For Markdown editors, e.g. Github infamous README.md or my blog posts, I used MacDown. After discovering Typora, MacDown becomes second in the pecking order for now (at least until Typora’s developers announce their paid plans).

With Typora, I can directly write Markdown articles just like Microsoft Word. I can do ⌘ + / to instantly switch back and forth between Visual Editing and Source modes. Even better, I can adapt the Markdown render in Typora to be nearly indentical to my blog and therefore, experience a kind of semi-WYSIWYG Markdown editing.

$ brew cask install typora

Other nice-to-have tools

GNU Tools

$ brew install coreutils autoconf automake findutils gawk gcc gnu-sed grep

Ack

Ack is faster and easier to use than grep

$ brew install ack

Dropbox

Many of our customers and friends are sharing files using Dropbox, so are we ;). Note that Dropbox app must be installed in /Applications to work properly.

$ brew cask install --appdir="/Applications" dropbox

Docker CE and command line

Every serious developer is talking about and working with it ;)

$ brew cask install docker
$ brew install docker

Spectacle

This is an amazing free app for resizing windows using shortcuts. I use it a lot to quickly arrange the working windows.

$ brew cask install spectacle

Alfred + Powerpack ($)

Alfred is a must-have for me and, maybe, many other macOS users. The free version is useful enough but the paid (for Powerpack) can blow anyone away. Alfred helps me much more when I prefer to use the keyboard as much as possible instead of reaching for the mice.

$ brew cask install alfred

F.lux

Saving my eyes staring daily at computer’s monitors (macOS also has Night Shift mode but my eyes still find F.lux more pleasant at night)

$ brew cask install flux

Web Browsers

Installing Firefox and Google Chrome mainly for Web development, testing and debugging as I find Safari is, not perfect though, sufficient for daily use and conveniently sharing bookmarks and history links between macOS and my iPad and iPhone.

$ brew cask install google-chrome firefox

Remote Access

Nothing is more convenient than SSH and SFTP.

# I prefer OpenSSH to the stock BSD SSH
$ brew install openssh
# ...  love to quickly copy my public key to the servers
$ brew install ssh-copy-id
# generate key pair
$ ssh-keygen -t rsa -b 4096 -C "MacBookPro email@example.com"
# copy the public key to the host for passwordless authentication
$ ssh-copy-id user@host

Networking Tools

$ brew install netcat curl wget nmap

Communication

Skype, Slack, Keybase (open source end-to-end encryption)

$ brew cask install skype slack keybase

1Password ($)

Help storing all of my private information and authentication in iCloud or Dropbox and using them in macOS or iOS.

$ brew cask install 1password

A journey from Apache CXF 2.2 to 3.2

Fri, 15 Jun 2018 00:00:00 +0000

I have used Apache CXF 2.2 to develop Web services for some R&D projects that I took part in. At that time, it was a choice between Apache Axis/Axis2 and CXF (formerly Codehaus XFire project).

I eventually decided to get along with CXF due to its simplicity, quite clear documentation, good support for document-style Web services (+) and many standards, especially JAX-WS and JAX-RS.

Moreover, Apache CXF also embraces smooth integration with Spring Framework (big plus for me as I was using Spring Web MVC 3 to develop the Web front-end). Everything went well for me on modelling and developing Web services based on JAX-WS for both directions: WSDL-first and Java-first.

In my projects, I designed the WSDLs and used WSDL2Java Maven plugin or command line to generate skeleton Java code of the Web services. The Java service implementations were kept separately in a package/folder to avoid any code overwriting. CXF services are so easy to configure with Spring Framework 3.0. The implementations of the services were loaded as Spring managed beans. The expose of a service is done via a CXF directive <jaxws:endpoint> of which the implementor attribute refers to the corresponding bean.

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:jaxws="http://cxf.apache.org/jaxws">
  <import resource="classpath:META-INF/cxf/cxf.xml" />
  <import resource="classpath:META-INF/cxf/cxf-extension-soap.xml" />
  <import resource="classpath:META-INF/cxf/cxf-extension-jaxws.xml" />
  <import resource="classpath:META-INF/cxf/cxf-servlet.xml" />    
  <bean id="CreditWorthinessImpl"
        class="westbank.ws.impl.CreditWorthinessImpl"
        p:dataAccessObject-ref="dataAccessObject">
  </bean>
  <jaxws:endpoint id="CreditWorthiness"
                  implementor="#CreditWorthinessImpl" 
                  address="/CreditWorthiness">
  </jaxws:endpoint>
</beans>

In the Web app configuration web.xml, CXF Servlet must be loaded by the container.

<servlet>
  <servlet-name>cxf</servlet-name>
  <servlet-class>org.apache.cxf.transport.servlet.CXFServlet</servlet-class>
  <load-on-startup>2</load-on-startup>
</servlet>
<servlet-mapping>
  <servlet-name>cxf</servlet-name>
  <url-pattern>/services/*</url-pattern>
</servlet-mapping>

Upgrading Apache CXF

For a while I haven’t taken part in further WS development, i.e. not using Apache CXF a lot. One day I thought I could use the WS project to showcase and review my relevant knowledge. The codebase still works well after some mysterious library missing that I have not seen before. It is possible due to the built-in libraries for XML parsing that were implicitly used but now changed in the newer JRE. Apart from that, most of the libraries are also outdated. So I dedice to spend some time to exercise upgrading the project and refactor a bit its source code. Here I jot down some major points on upgrading Apache CXF.

The upgrading was not as easy and smooth as just changing the dependencies’ versions. A lot of conflicts or major refactoring happened. I had to read thoroughly the documentation on CXF site for migration and decided to go gradually over each major version of Apache CXF.

… from version 2.2 to 2.7 …

From Apache CXF 2.2 to 2.6, not so many changes are relevant for the project as I mainly used cxf-rt-frontend-jaxws. Another significant change since 2.4 causes errors for Spring / Jetty server regarding importing CXF’s XML resources. Recall the aforementioned Spring bean configuration where CXF service implementations were loaded and published? There are a number of <import> directives. These directives advise Spring to load the corresponding CXF resources. Now we only need “one to rule them all”.

<import resource="classpath:META-INF/cxf/cxf.xml" />

To upgrade to version 2.6, I have to revise the project with respect to the merging of cxf-common-utilites into cxf-api, the removal of cxf-rt-binding-http, and the refactoring that impact cxf-rt-core. Most of the other changes from 2.2 to 2.7 are for JAX-RS, which were used very little or none in my project.

… and to version 3.0+

Apache CXF 3.0 requires a rather disruptive change, from Spring Framework 3.0 to 3.2+. Again, I used standard Spring bean configurations which are still valid for Spring 3.2. Hence, the project works well with Spring 3.2.18-RELEASE. I only need to remove all version numbers in the Spring XML schemas. The major change I must deal with is to remove the dependency of cxf-api as it was merged with cxf-core.

Then there comes the most dramatic issue with CXF 3.0.16 (sic!). cxf-codegen-plugin used to generate Java code from WSDLs refused to work (which was no problem in earlier verions) and spit out errors like this.

[ERROR] Failed to execute goal org.apache.cxf:cxf-codegen-plugin:3.0.16:wsdl2java (generate-sources) on project loan-approval-portal: Execution generate-sources of goal org.apache.cxf:cxf-codegen-plugin:3.0.16:wsdl2java failed: org.apache.cxf.wsdl11.WSDLRuntimeException: Fail to create wsdl definition file:%3c?xml%20version=%221.0%22%20encoding=%22UTF-8%22?%3e: WSDLException: faultCode=PARSER_ERROR: Problem parsing 'file:%3c?xml%20version=%221.0%22%20encoding=%22UTF-8%22?%3e'.: java.io.FileNotFoundException: < (No such file or directory) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.cxf:cxf-codegen-plugin:3.0.16:wsdl2java (generate-sources) on project loan-approval-portal: Execution generate-sources of goal org.apache.cxf:cxf-codegen-plugin:3.0.16:wsdl2java failed: org.apache.cxf.wsdl11.WSDLRuntimeException: Fail to create wsdl definition file:%3c?xml%20version=%221.0%22%20encoding=%22UTF-8%22?%3e: WSDLException: faultCode=PARSER_ERROR: Problem parsing 'file:%3c?xml%20version=%221.0%22%20encoding=%22UTF-8%22?%3e'.: java.io.FileNotFoundException: < (No such file or directory)

What puzzles me is that I have tried the command tool wsdl2java of Apache CXF 3.0.16 with each WSDL and, strangely, found no errors at all. After some extra trial-and-error effort, I eventually figured out that an extra option for cxf-codegen-plugin causes the error. The old/original plugin configuration in pom.xml is as following.

<plugin>
  <groupId>org.apache.cxf</groupId>
  <artifactId>cxf-codegen-plugin</artifactId>
  <version>${cxf.version}</version>
  <executions>
    <execution>
      <id>generate-sources</id>
      <phase>generate-sources</phase>
      <configuration>
        <sourceRoot>${basedir}/src/main/java</sourceRoot>
        <wsdlRoot>${basedir}/src/main/webapp/WEB-INF/wsdl/</wsdlRoot>
        <defaultOptions>
          <validateWsdl>true</validateWsdl>
          <wsdlList>true</wsdlList>
          <defaultExcludesNamespace>true</defaultExcludesNamespace>
          <extraargs>
            <extraarg>-defaultValues</extraarg>
            <extraarg>-quiet</extraarg>
            <extraarg>-wsdlLocation</extraarg>
            <extraarg></extraarg>
          </extraargs>
        </defaultOptions>
      </configuration>
      <goals>
        <goal>wsdl2java</goal>
      </goals>
    </execution>
  </executions>
</plugin>

The option <wsdlList> is the culprit. It was declared boolean since CXF 2.2 to 2.7.0 as I found here. However, in the most recent documentation of CXF 3, wsdlList is still listed as an option for wsdl2java but no longer of type boolean (-wsdlList <wsdlurl>). To make it worse, the option is totally hidden/undocumented. So all I have to do is to disable the line <wsdlList>true</wsdlList> and cxf-codegen-plugin works again in CXF 3.0+.

The problems with cxf-codegen-plugin keep raising when upgrading CXF to 3.2.4. This time, it threw another exception though.

[ERROR] Failed to execute goal org.apache.cxf:cxf-codegen-plugin:3.2.4:wsdl2java (generate-sources) on project loan-approval-portal: Execution generate-sources of goal org.apache.cxf:cxf-codegen-plugin:3.2.4:wsdl2java failed: org.xml.sax.SAXNotRecognizedException: Property 'http://javax.xml.XMLConstants/property/accessExternalSchema' is not recognized. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.cxf:cxf-codegen-plugin:3.2.4:wsdl2java (generate-sources) on project loan-approval-portal: Execution generate-sources of goal org.apache.cxf:cxf-codegen-plugin:3.2.4:wsdl2java failed: org.xml.sax.SAXNotRecognizedException: Property 'http://javax.xml.XMLConstants/property/accessExternalSchema' is not recognized.

We can see that the error is due to schema validation. Before, I wanted to make sure that the WSDLs were valid before generating Java code. Thus, I chose to enable validation by <validateWsdl>true</validateWsdl>. This option is actually the root cause of the aforementioned exception during validation. The problem seems to stem from new XML security properties in JAXB 1.5 introduced in Java 8. When I disable that option, the plugin works fine again (voila!). But that means I must live in a maybe-not-error-free world (D’oh!). But we are all, aren’t we?

After all of the above, I can get my project to work with Apache CXF 3.2.4 while keeping the business logic of services intact.

Update 2018-06-25: XML-less Spring Configuration

As deciding to switch to XML-less Spring configuration to learn more about Spring Java annotations, I also tried to migrate CXF settings, too. Here are some last updates.

Loading CXFServlet

public class MyWebApplicationInitializer implements org.springframework.web.WebApplicationInitializer {
  @Override
  public void onStartup(ServletContext container) throws ServletException {
    AnnotationConfigWebApplicationContext root = new AnnotationConfigWebApplicationContext();
    root.register(ServiceConfiguration.class);
    root.refresh();

    // Register and map the CXF servlet
    CXFServlet cxfServlet = new CXFServlet();
    ServletRegistration.Dynamic reg = container.addServlet("cxf", cxfServlet);
    reg.setLoadOnStartup(1);
    reg.addMapping("/services/*");
  }
}

Publishing Services

I configured the CXF bus used for publishing web services and defining JAX-WS service endpoints. Instead of importing existing CXF’s XML resources as many developers have chosen, I opted for a pure Java approach. What we have to do is to create a bean named ‘cxf’ (defined as org.apache.cxf.Bus.DEFAULT_BUS_ID) and use that bean for defining service endpoints.

import com.example.HelloWorldImpl;

import org.apache.cxf.Bus;
import org.apache.cxf.bus.spring.SpringBus;
import org.apache.cxf.jaxws.EndpointImpl;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.xml.ws.Endpoint;

@Configuration
public class ServiceConfiguration {

  @Bean(name=Bus.DEFAULT_BUS_ID)
  public SpringBus cxf() {
      return new SpringBus();
  }

  @Bean
  public Endpoint helloWorld() {
    HelloWorldImpl implementor = new HelloWorldImpl();
    EndpointImpl endpoint = new EndpointImpl(cxf(), implementor);
    endpoint.publish("/HelloWorld");
    return endpoint;
  }
}

IMPORTANT Endpoint.publish() must be called to expose the service endpoint instead of Endpoint.setAddress() as above.

Integration Laravel 5 and Gentelella admin theme

Wed, 09 May 2018 00:00:00 +0000

Gentelella is a very nice and gently-looking Bootstrap based admin theme developed by Aigars Silkalns (aka puikinsh). It can be integrated into several Web application frameworks. This post is one of my development exercises in which Gentelella is used to decorate a Laravel 5 based Web application. This task is part of building my PHP based framework LaraMod.

Laravel can act as backend for any Web application. At the moment of this writing, Laravel includes several good supports for popular front-ends such as Bootstrap, Vue.js, and React. The included tool Laravel Mix, which runs on top of Webpack, can significantly lessen development effort on compiling and mixing Web resources. Thus, I have just leveraged Laravel Mix for this integration task. Further details on Laravel Mix can be found here.

Installation

Laravel provides built-in support for some front-end frameworks but they might cause some conflicts and incompability with Gentelella’s own libraries. Therefore, I have decided to start from scratch, i.e. using basic Laravel without any front-end scaffolding

# remove all scaffolding
php artisan preset none

This command will remove all relevant scaffolding components including Bootstrap, Vue or React. Note that Laravel configuration for Bootstrap and Vue might still exist. For instance, check resource/assets/js/app.js for the following line and make sure it is commented out.

// require('./bootstrap');

Now we just add Gentelella’s dependencies.

# adding Gentelella
yarn add --save gentelella 		# or npm install gentelella --save
# update dependencies
yarn install					# or  npm install

After this step, Gentelella should be available in the local node_modules folder.

Compiling and Mixing Resources

The main configuration file of Laravel Mix is webpack.mix.js. The fundamental tasks are defined as following.

let mix = require('laravel-mix');
...
mix.js('resources/assets/js/app.js', 'public/js')
   .sass('resources/assets/sass/app.scss', 'public/css');

That is, Mix will compile JavaScript and SASS resources in resources/assets/js/app.js and resources/assets/sass/app.scss to public/js and public/css, respectively.

We shall use the same way to compile necessary Gentelella resources. The full configuration can be found in the LaraMod project webpack.mix.js.

Some common variables are defined to reduce duplicates.

// general resources
let public_js = 'public/js/';
let public_css = 'public/css/';
let resource_sass = 'resources/assets/sass/';
// Gentelella resources
let gentelella_home = 'node_modules/gentelella/';
let gentelella_vendor = gentelella_home + '/vendors/';

As using separate styles for different pages, e.g. Login, Dashboard, etc., I configure Mix to compile them accordingly.

mix.js('resources/assets/js/app.js', public_js).
    sass(resource_sass + 'app.scss', public_css).
    sass(resource_sass + 'home.scss', public_css).
    options({processCssUrls: false}).
    sass(resource_sass + 'login.scss', public_css).
    options({processCssUrls: false}).
    sass(resource_sass + 'dashboard.scss', public_css).
    options({processCssUrls: false});

For each component of Gentelella that we need for our project or certain pages, we can copy them into Laravel public folders.

/*
 *  Copy dependent JavaScripts and CSSs
 */
mix.
    // gentelella
    copy(gentelella_home + 'build/css/custom.css',
        public_css + 'gentelella-custom.css').
    copy(gentelella_home + 'build/js/custom.js',
        public_js + 'gentelella-custom.js').
...

We can definitely use Mix/Webpack tools for optimising/minimising/transforming Gentelella resources. I will leave it as extra deployment exercises. For now, I will mainly concentrate on development aspects. When I need to use a Gentelella’s component for a certain page, I will include the following directives in the corresponding Laravel Blade templates for that page.

<!-- Styles -->
<link href="{{ asset('css/font-awesome.css') }}" rel="stylesheet">
<link href="{{ asset('css/bootstrap.css') }}" rel="stylesheet">
<link href="{{ asset('css/gentelella-custom.css') }}" rel="stylesheet">
...
<!-- Scripts -->
<script src="{{ asset('js/jquery.min.js') }}"></script>
<script src="{{ asset('js/bootstrap.js') }}"></script>
<script src="{{ asset('js/gentelella-custom.js') }}"></script>
<script src="{{ asset('js/app.js') }}"></script>

Note that the specific Laravel Blade directive {{ }} and asset(...) are used to refer to the resources inside Laravel public/ folder accordingly. A working example of Laravel and Gentelella integration is illustrated via the LaraMod project that you can check out and have a deeper look.

Conclusion

Integration Laravel 5 and Gentelella is a simple but interesting exercise showing many nice features of Laravel for supporting common front-ends. Laravel Mix provides a good wrapper on top of Webpack that really reduces the complexity of Webpack and lessens considerable amount of development effort.

Node.js package managers: NPM and Yarn

Mon, 23 Apr 2018 00:00:00 +0000

Node.js is an increasingly popular and widely used JavaScript runtime nowadays. One of the important aspects of Node.js is to manage the dependencies among software artefacts. Better dependency management would advocate modularisation and decoupling of software components. Since the dawn of Node.js, several package managers have been developed, notably, npm, anymod (formerly component), volo, ringojs, bower, yarn, pnpm, to name but a few.

In this post, we shall walk through the two tools that have been seemingly living up to the high expectation of a majority of JavaScript developers and communities, npm and yarn. pnpm is an improvement that performs sometimes better than npm and yarn. The great advantage of using pnpm is that we just replace the command npm by pnpm and keep the rest intact. Bower is a powerful tool that supports not only JS but also various types of Web resources including HTML, CSS, fonts, images. Nevertheless, the core Bower developers and maintainers had recently recommended the users to switch to yarn.

npm seems to be the de facto package management included with Node.js. When Node.js is installed, npm will also be available and ready to use as well. This is one among many reasons why npm is well-known and widely used by JS developers.
yarn was originally developed by Facebook to overcome existing problems of existing package management tools. Some notable yarn’s features are deterministic model (producing the same result when repeating), flat mode (resolving mismatching versions of dependencies to a single version to avoid duplicate), security first, offline mode, network performance, and so forth. Nevertheless, recent versions of npm also improve significantly by learning and incorporate several good features from yarn.

Walking Through

In this part, we will go through a typical development workflow that each tool provides with basic steps and some commonly used options.

0. Installing and Upgrading

NPM

As mentioned above, npm is available wherever Node.js is installed. So, the only thing to do is to upgrade npm

npm install npm@latest -g
# you can also use the alias 'i'
npm i npm@latest -g

Yarn

Yarn can be installed via npm but this method is not recommended due to security reason.

npm install --g yarn

macOS users can install Yarn using Homebrew.

# installing
brew update && brew install yarn
# upgrading
brew upgrade yarn

There are also concrete instructions for Linux and Windows users, too.

1. Starting New Projects

NPM

npm init [-f|--force] [-y|--yes]

npm init will ask you a number of questions and create an initial configuration file package.json. In case you do not want to answer the questions one by one, you can use any option -f, --force, -y or --yes to skip all questions and get a default package.json in the current directory.

Yarn

yarn init [-y | --yes] [-p | --private]

This command is totally similar to npm init and the outcome will be package.json. The only difference is -p or --private to set "private": true.

2. Getting Information

NPM

# display a particular package's information
npm view eslint
npm info eslint
# search for a package/name
npm search eslint
# list installed packages
npm ls

Yarn

Yarn developers deliberately do not add support for searching packages like npm as explained here.

# display a particular package's information
yarn info eslint
# list installed packages
yarn list

Yarn provodes a nice command for showing why a certain package was installed.

yarn why eslint

3. Installing Dependencies

Both npm and yarn provide several options to add dependencies ranging from registered packages, tarballs, to git repositories. Please note that, npm and yarn also support manipulating global package repositories, i.e. installed packages that are available to the whole working system instead of local projects. In most of the case, we can use the option -g for npm and the command global for yarn. Thus, in the following steps, we mainly concentrate on local repositories.

NPM

# install all dependencies defined in package.json
npm install
# add a registered package and record it in 'dependencies' 
npm install eslint
# add an exact version
npm install eslint@3.0.0
# add and record in 'devDependencies'
npm install --save-dev eslint
# add a git repos
npm install git@github.com:eslint/eslint.git

Yarn

# install all dependencies defined in package.json
yarn install
# add a registered package
yarn add eslint
# add an exact version
yarn add eslint@3.0.0
# add to dev dependencies
yarn add --dev eslint
# add a git repos
yarn add git@github.com:eslint/eslint.git

4. Upgrading Packages

NPM

# check outdated packages
npm outdated
# upgrade all to the latest versions w.r.t version ranges in 'package.json' (since 2.6.1 default to top level packages)
npm update
# upgrade a specific package
npm update eslint

Yarn

# check outdated packages
yarn outdated
# upgrade all
yarn upgrade
# upgrade all and ignore version ranges in 'package.json'
yarn upgrade --latest
# upgrade a specific package
yarn upgrade eslint
# upgrade packages match a pattern
yarn upgrade --pattern eslint

5. Removing Packages

NPM

npm uninstall eslint

Yarn

yarn remove eslint

6. Running Commands

NPM

npm allows users to define executable scripts in package.json under the section

"scripts" : { "test" : "..." }

then use npm run script-name to execute the predefined scripts.

npm run test

Nevertheless, npm provides a short form for executing testing scripts as well.

npm test

Yarn

Similar to npm, yarn users can execute scripts with yarn run script-name and yarn test for testing.

yarn run start
yarn test

So far, we have walked through some basic steps of a typical development workflow, ranging from initialisation to inquiring and manipulating packages. Apart from that, both npm and yarn also go extra length with many more different functionality. Backing by very strong and active communities, your development projects will surely in safe hands when choosing either of them.

Confusing Hugo issues: datetime and 'isset'

Sat, 31 Mar 2018 00:00:00 +0000

Yesterday I got notified from Github for two opening issues of the Hugo’s theme hyde-hyde I have ported and developed further.

The newest one is reported by jdayton3 (Jonathan Dayton) on wrong dates shown up. The issue seems very strange and difficult to spot. It took me a while to research around and found some relevant reports by Paul Heinlein and Dana Woodman, that Hugo date/time formatting is internally based on Golang, and therefore, uses a smart but confusing convention. It’s funny that I made almost the same mistake as Dana (i.e. he used the format string “March 1, 2010” whilst mine is “Jan 1, 2006”). As such, the generated dates were totally incorrect.

Paul’s post summarise excellently the Go’s date/time format convention:

Format strings absolutely must adhere to the 1-2-3-4-5-6-7 order:

Month must be Jan, January, 01, or 1

Date must be 02 or 2

Hour must be 03, 3, or 15

Minute must 04

Second must be 05

Year must be 2006

Timezone must be MST or -7

After changing the format string, the aforementioned issue surely disappears. Even better, hyde-hyde date/time formatting can also now adopt a default option to prevent such potential issues.

{{ .Date.Format (.Site.Params.dateformat | default "Jan 02, 2006") }}

The other issue reported by paskal (Dmitry Verkhoturov) is an error Dmitry spotted when compiling and testing his blog against several Hugo themes including mine, hyde-hyde.

Building sites … ERROR 2018/03/20 20:35:56 Error while rendering "page" in "": template: /srv/hugo/themes/hyde-hyde/layouts/_default/single.html:6:7: executing "content" at <partial "post_conten...>: error calling partial: template: theme/partials/post_content.html:22:21: executing "theme/partials/post_content.html" at <len .Params.tags>: error calling len: len of untyped nil
ERROR 2018/03/20 20:35:56 Error while rendering "page" in "post/": template: /srv/hugo/themes/hyde-hyde/layouts/_default/single.html:6:7: executing "content" at <partial "post_conten...>: error calling partial: template: theme/partials/post_content.html:22:21: executing "theme/partials/post_content.html" at <len .Params.tags>: error calling len: len of untyped nil

The problem related to accessing the tags of each post via .Params.tags. The function len .Params.tags receives an untyped nil value. Strangely, there is a conditional check using the function isset right before that.

{{ if isset .Params "tags" }}
   {{ $total := len .Params.tags }}
   ...
{{ end }}

The semantics of isset is a bit vague:

Returns true if the parameter is set.

isset COLLECTION INDEX
isset COLLECTION KEY

It does not say anything when either COLLECTION or INDEX/KEY is nil or empty. As such, perhaps Dmitry’s blog contains some posts with empty tags. Hence, the variable .Params.tags yields "" whilst isset.Params "tags" evaluates to true.

One of the temporary mitigation is to use with instead. The semantics of with is a bit clearer.

Rebinds the context (.) within its scope and skips the block if the variable is absent.

Nevertheless, the error is dismissed when replacing the above code with the following. Hoooraaay!!!

{{ with .Params.tags }}
	{{ $total := len . }}
{{ end }}

Hugo is quite tricky, eh?

Getting started with Webpack

Fri, 16 Feb 2018 00:00:00 +0000

Webpack is a static module bundler for Web applications. It analyses and processes the input application and generates output bundles.

The design of Webpack architecture is highly modularised and extendable. In this post, I share here some notes on major aspects of Webpack since I’ve started learning it.

Figure 1. Webpack overview [source]

Main Concepts

Configuration

Webpack’s configuration is conventionally defined using a file webpack.config.js. It is a JavaScript/Node.js source file. A comprehensive example of webpack.config.js can be found here. The four major parts of Webpack’s configuration are entry, output, loaders and plugins.

Entry: An entry denotes the starting point where Webpack commences buidling the internal dependency graph by analysing all direct and indirect dependencies. There might be more than one entry.
Output: The output part defines the places where Webpack stores and how to name the outputs.
Loaders: Webpack uses loaders to handle files other than JavaScript by transform them into modules that can be processed by Webpack. Loaders are configured using module.rules that specify the properties test (what/which files to be transformed) and use (which loaders to be used).
Plugins: Plugins are important part of Webpack that will carry out different kinds of tasks such as checking, combining files, optimised outputs, and so on. Webpack provides a clear interface for creating and/or extending plugins.

Modules

Webpack considers almost each file of any kinds a module. The dependencies between modules can be described via various ways, for instance using ES2015’s import, CommonJS’s require(), AMD’s define and require, and CSS’s @import. Some built-in module types supported by Webpack are including CoffeeScript, TypeScript, Babel, Sass, Less, and Stylus.

Dependency Graph

When a file or module needs another, it is considered a dependency. Webpack analyses all possible dependencies and builds a graph that includes all needed modules starting from the entry points. The dependency graph is then used to package these modules into the output bundles.

Some Typical Usage Scenarios

1. A Simple Greeting

To demonstrate the simplest and, somewhat naive, usage of Webpack, we will create a small project as following.

simple-greeting
 ├── package.json
 ├── public
 │   └── index.html
 ├── src
 │   └── main.js
 └── webpack.config.js

The package.json can be quickly instantiated using npm or yarn, as I prefer)

$ cd simple-greeting
# create a package.json with default options
$ yarn init -y
# now we use npm install webpack as a dependency of our project
$ yarn add --dev webpack

Let’s create a JavaScript src/main.js that contains our main business logic, i.e. writing out a heading 1 Hello Webpack!.

document.write('<h1>Hello Webpack!</h1>');

And we edit public/index.html.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Getting Started with Webpack - A Simple Greeting</title>
</head>
<body>
    <script src="bundle.js"></script>
</body>
</html>

You might have noticed that index.html uses bundle.js instead of main.js that we created above. You are right, bundle.js is generated by Webpack given the input main.js. Let’s create a config file webpack.config.js to do that.

const webpack = require('webpack');
const path = require('path');

module.exports = {
  entry: './src/main.js',
  output: {
    path: path.resolve(__dirname, 'public'),
    filename: 'bundle.js',
  }
};

After that, a simple invocation of webpack at the command line will work. Please note that npm installs the executable webpack inside node_modules/.bin.

$ node_modules/.bin/webpack
Hash: 9300b893968675cae1ef
Version: webpack 3.11.0
Time: 81ms
    Asset     Size  Chunks             Chunk Names
bundle.js  2.52 kB       0  [emitted]  main
   [0] ./src/main.js 43 bytes {0} [built]

The execution of webpack command should be successful. Now should you open public/index.html using a Web browser, you will see “Hello Webpack”.

2. Using Loaders

Several loaders have been developed for Webpack in order to handle various application resource types. An incompleted list of Webpack loaders can be found here. We will exemplify eslint-loader for analysing JavaScript sources and reporting errors, if any.

Let’s reuse the previous example webpack.config.js and add the eslint-loader.

module.exports = {
    entry: './src/main.js',
    output: {
        path: path.resolve(__dirname, 'public'),
		filename: 'bundle.js'
    },    
    module: {
        rules: [
            {
                test:/\.js$/,
                exclude:/node_modules/,
                loader: 'eslint-loader'
            }
        ],   
    },
};

We also needs, at least, the following packages: eslint and eslint-loader which can be easily installed. In case you need further styles or rules, you can install more packages, for instance, eslint-config-airbnb, eslint-config-google, to name but a few.

$ yarn add --dev eslint eslint-loader

For starting, we can initialise a simple configuration for eslint using the option --init and answer the corresponding questions. eslint will create a file .eslintrc.xxx where “xxx” is either “js”, “json”, or “yaml” depending on which file format you had chosen.

$ eslint --init
...

Then we can invoke webpack to trigger the loader. For example, we can add the following line in the src/main.js

document.write('<h1>Hello Webpack!</h1>');
document.write(x); // eslint will report an error here

$ webpack
Hash: 9b7cb596310bf077e30b
Version: webpack 3.11.0
Time: 622ms
    Asset     Size  Chunks             Chunk Names
bundle.js  2.53 kB       0  [emitted]  main
   [0] ./src/main.js 61 bytes {0} [built] [1 error]

ERROR in ./src/main.js

/Users/huytran/working/dev/dev-web/webpack-demo/using-linter/src/main.js
  2:16  error  'x' is not defined  no-undef

✖ 1 problem (1 error, 0 warnings)

So you can see that eslint-loader indeed caught the intended error. All errors should be fixed such that Webpack can move forward.

3. Using Plugins

As mentioned above, Webpack architecture enables the use of plugins for performing various kinds of tasks, for example, merging, minimising or uglifying source code. Many of Webpack plugins and guides can be found here or here. We take an example of UglifyjsWebpackPlugin to illustrate how plugins actually work.

Let’s modify the file webpack.config.js taken from the previous scenarios and add the plugins part.

module.exports = {
    entry: './src/main.js',
    output: {
        path: path.resolve(__dirname, 'public'),
		filename: 'bundle.js'
    },
    ...
    plugins: [
        new webpack.optimize.UglifyJsPlugin()
    ]
};

After executing node_modules/.bin/webpack again, you can see that JavaScript source in public/bundle.js has been uglified/minimised whilst the ouput of public/index.html remains in tact.

So far, we have walked through some simple scenarios. They are not quite complex and might have not yet shown all aspects and power of Webpack. Nevertheless, I hope they are sufficient for just showing the basis of Webpack architecture and how it works.

The aforementioned demo projects are hosted at Github, respectively.

First experience with Hugo

Tue, 23 Jan 2018 00:00:00 +0000

To commence my journey in software development, I have looked for a Web framework to build my personal blog. I have started writing blog posts mainly using (Multi)Markdown and plan to migrate other posts to Markdown too. Thus, it’s rather natural to go with built-in or first-class support for Markdown to minimise the migration effort.

I have not foreseen any great use of extensive backend storage and computation thus far. Therefore, I narrowed down my search to static site generators with two prominent candidates Jekyll and Hugo popping up, for instance, here, here, and here. Both offer all essential features to bootstrap a new site or migrate existing sites. Nothing can go wrong when you pick either one.

After a week trying both Jekyll and Hugo, I ended up deciding to go with Hugo. It’s not to say I hate Jekyll but quite the opposite. Jekyll is a rather well-engineered piece of software (by the guys behind Github). I love its design, strong community, rich ecosystem, a lot of beautiful themes and good integration with Github.

There is just one thing I do not feel comfortable working with Jekyll is the way to apply a theme. As far as I know, most of Jekyll’s themes ask for copying and pasting theme resources (e.g. CSS, templates) into an existing Jekyll site (please correct me if I am wrong).

This is quite strange regarding the conventional approach of Ruby – the backend of Jekyll. As predicted, I had way too many problems changing themes that cross various Jekyll versions. It should be the other way around. That is, a theme would remain in tact inside predefined folders and be plugged into the Jekyll site via the configuration file. I think this approach, in terms of software design, would be less intrusive and much lower coupling.

Hugo gets this design decision quite right. That’s why I prefer Hugo, apart from its amazing performance. You can easily find numerous compliments for Hugo around the Internet. I just fall in love with the way Hugo allows me to apply and adapt any themes (of course with some extra effort as each theme might often introduce some additional customizable features or variables). Besides, as I desire to learn more about Golang, Hugo would be definitely a nice start.

The longer I play around with Hugo, the more I become passionate about Hugo in particular and Golang in general. The first week working with Hugo has yielded a Hugo’s theme for my personal use that you might find useful, namely, hyde-hyde (based on spf13’s Hyde). My journey with Hugo and Golang has just started but it looks quite bright and promising thus far.

I ❤️ Hugo.

LaraMod - Modularised Laravel 5

Fri, 19 Jan 2018 00:00:00 +0000

Laravel is an amazing emerging, well-designed and well-developed PHP framework. It is currently under active development and has been among top Web development frameworks. For a simple development project, the structure is totally fine. Nevertheless, you might not want to mix up your source code and resources with Laravel’s except some really necessary configurations.

Moreover, you might also want to organise your project into submodules that are assigned to more than one team/person such that they can be developed silmutaneously. Thus, the need for complex submodule organisation is quite inevitable.

From my struggle to organise a Laravel-based project such that I can divide and work on individual submodules such as UserManagement, Authentication, Dashboard, etc. whilst keeping the Laravel code base intact as much as possible (this could be convenient for upgrading Laravel) and keeping my code base separate from Laravel’s.

There are a number decent attempts on modalurasing Laravel projects for example here, here or here. Being inspired and learning from these articles, I decided to start a simple project on my own, namely, LaraMod, for many reasons, but the biggest one is to dig deeper into Laravel 5 and PHP.

Here I only emphasize some major aspects of LaraMod. The rest, including code and extra improvements can be found at LaraMod’s repository.

Update 1 @ 2018-04-25: LaraMod has been revised and updated to work with the most recent version of Laravel framework, 5.6.17. Instead of a standalone git repos, LaraMod is from now on a fork of Laravel repos enhanced with better modularisation.

Update 2 @ 2018-11-30: LaraMod has been upgraded and merged with Laravel master in which the stable release version is 5.7.15.

Modularising Laravel 5

A Simple Submodule Structure

Assuming that I want to divide my project into submodules of which each comprises own controllers, views, models (MVC) and others such as database migration, i18n, and routes. A simple structure of the submodule Authentication is shown as following.

Authentication
├──Controllers
├──Lang
├──Migrations
├──Models
├──View
└──routes.php

Such a structure can be very useful for different teams/people working independently. In LaraMod, there is an artisan command gen:module provided in App\Console\Commands\GenModuleCommand that can quickly create a submodule following that structure.

$ php artisan gen:module Authentication

How to Proceed

An easy approach is to rely on Laravel’s ServiceProvider to load the submodules and register the necessary components of the submodules.

<?php
class XYZServiceProvider extends \Illuminate\Support\ServiceProvider {
  public function boot() {}
  public function register() {}
}

A ServiceProvider contains two important methods register() and boot(). In this case, LaraMod mainly uses the method boot() to load the corresponding submodule. LaraMod also adopts a simple, conventional method for defining a submodule. That is, all submodules will be subfolders of the folder /modules and each module follows the conventional structure as mentioned above.

Automatically Loading Submodules

Loading a Submodule

First, we create a method loadModule() to load the resources such as views, i18n, database migration, and routes of a submodule using the provided methods loadViewsFrom(), loadTranslationsFrom(), loadMigrationsFrom(), loadRoutesFrom(), respectively. As models and controllers are essential PHP classes, they can be loaded using PSR-4 autoloaders as shown in the next parts.

<?php
protected function loadModule($module)
{
    if ($module) {
      // load the submodule's routes
        if (file_exists(__DIR__ . '/' . $module . '/routes.php')) {
            $this->loadRoutesFrom(__DIR__ . '/' . $module . '/routes.php');
        }
      // load the submodule's views
        if (is_dir(__DIR__ . '/' . $module . '/Views')) {
            $this->loadViewsFrom(__DIR__ . '/' . $module . '/Views', $module);
        }
      // load the submodule's translation
        if (is_dir(__DIR__ . '/' . $module . '/Lang')) {
            $this->loadTranslationsFrom(__DIR__ . '/' . $module . '/Lang', $module);
        }
      // load the submodule's database migrations
        if (is_dir(__DIR__ . '/' . $module . '/Migrations')) {
            $this->loadMigrationsFrom(__DIR__ . '/' . $module . '/Migrations', $module);
        }
    }
}

Detecting and Loading Submodules

Then we can walk through the folder module and load all submodules using the method loadModule() created above.

<?php
protected function findAndLoadModules()
{
    $modules = config("module.modules");
    if (!$modules) {
        $modules = $this->getModuleNamesFromCurrentPath();
    }
    if ($modules) {
        array_walk($modules, function ($module) {
            $this->loadModule($module);
        });
    }
}

Note that in this conventional approach, findAndLoadModules() will look for submodule configurations either (1) explicitly in the file /config/module.php with the following syntax or (2) implicitly as subfolders of /modules (in case /config/module.php does not exist).

// module.php
<?php
return  [
    'modules' => [
        'Core',
        'Login',
        'Dashboard',
        'Home',
        'User',
    ]
];

Finally we invoke findAndLoadModules() within boot().

<?php
namespace Laramod;
class ModulesServiceProvider extends \Illuminate\Support\ServiceProvider
{
    public function boot()
    {
        $this->findAndLoadModules();
    }
    public function register()
    {
    }
}

Registering the Service Provider

The ModulesServiceProvider must be registered in /config/app.php in order to be loaded by Laravel.

<?php
return [
    ...
    'providers' => [
        ...
        \Laramod\ModulesServiceProvider::class,
     ],
    ...
];

Autoloading Submodules’ Classes

Apart from resources like views, i18n, database migration loaded in the previous steps, a submodule can also contain other PHP classes such as database ORM models and controllers.

One natural approach would be to leverage the PSR-4 autoloader autoloading supported by Laravel 5.5. This approach also nicely fits when we want to define separate namespaces for each submodule and especially submodule’s components.

For this, we can define the submodule’s namespace and manually specify the submodule’s classpaths to be loaded in composer.json as following.

{
  "psr-4": {
    "App\\": "app/",
    "Core\\": "modules/Core/",
    "Dashboard\\": "modules/Dashboard/",
    "Login\\": "modules/Login/",
    "User\\": "modules/User/"
  }
}

Another technique for autoloading classes in Laravel 5 is to dynamically add PSR4 classpaths in ModulesServiceProvider.php.

// ModulesServiceProvider.php
<?php
...
$loader = require base_path() . '/vendor/autoload.php';
$loader->setPsr4("Core\\", __DIR__ . "./Core/");
$loader->setPsr4("Dashboard\\", __DIR__ . "./Dashboard/");
$loader->setPsr4("Login\\", __DIR__ . "./Login/");
$loader->setPsr4("User\\", __DIR__ . "./User/");

One problem, though, is that LaraMod must be able to detect or guess the namespace classpaths. To do this, we can either use the configuration file config/module.php or, even nicer and smarter, scan the submodule’s folders to extract the necessary information. However, loading time then will be really significant when the project grows because Laravel loaders willl do the scanning and loading whenever any classes are referenced.

Accessing Submodule Resources

Given a submodule, for instance SubmoduleA, most of its classes such as controllers and database mapping models, if configuring properly using PSR-4, will be automatically loaded and used straightforwardly. Nonetheless, acessing to resources such as views and i18n (languages) requires a slightly different syntax, for instance, SubmoduleA::blade_view_name or SubmoduleA::messages.error. If the SubmoduleA:: is missing, Laravel will look for views and language files in the default places.

Finale

The outcome of my very first PHP project, LaraMod, would be a reasonable skeleton for modularising software projects based on Laravel 5. That is, you can just easily create a new submodule with the predefined conventional structure and add the PSR-4 namespaces and classpaths, then good you go. Each submodule should then be developed independently. Sure there are some areas that need improvement. For example, all testing stuffs are still under /tests and database seeders are still in database/seeds. The first one requires intervention with PHPUnit/phpunit.xml whilst the later asks for further changes in built-in Laravel artisan commands.

For me—a new bie, it was not quite an all-pleasant journey to experience several Laravel aspects. Nevertheless, developing and customising LaraMod are extremely valuable as I could learn a lot about not only Laravel in particular but also PHP in general. Achieving reasonable modularisation for a complex software development project requires a lot of thoughtful design and hard work and even sometimes trade-offs and compromises. In sharing this, I hope LaraMod might become handy and helpful start so that your journeys with Laravel can be more pleasant and joyful. I look forward to hearing and learning from your experience, too.

Ngụy biện / Fallacies (st)

Mon, 01 Jan 2018 00:00:00 +0000

Bài viết sưu tầm trên Internet về các hình thức ngụy biện khác nhau.

Thay đổi chủ đề

Công kích cá nhân (ad hominem). Đây là một loại ngụy biện phổ biến nhất, nguy hiểm nhất, và có “công hiệu” nhất, vì nó tấn công vào cá nhân của người tranh luận, và tìm cách trốn tránh luận điểm của cá nhân đó. Hình thức ngụy biện này thường xuất hiện dưới dạng: Ông A phát biểu về một vấn đề; ông B tấn công vào cá nhân ông A, và làm cho người ta nghi ngờ luận điểm của ông A. Tuy nhiên, có thể không có mối liên hệ nào giữa cá nhân và luận điểm của ông A.

Có hai hình thức thuộc loại ngụy biện này. Thứ nhất là dưới hình thức sỉ nhục, hay chửi rủa. Khi bất đồng ý kiến, người ngụy biện chỉ việc công kích vào cá nhân của người phát biểu.

Hình thức ngụy biện thứ hai trong loại này là người ngụy biện cố gắng thuyết phục người đối thoại chấp nhận luận điểm của họ bằng cách đề cập đến hoàn cảnh của cá nhân đó. Ví dụ: “Anh nói là không nên uống rượu, vậy mà anh đã từng ngất ngưởng cả năm qua.”

Lợi dụng quyền lực (ad verecundiam). Đây là loại ngụy biện dùng những nhân vật nổi tiếng hay được nhiều người ái mộ để tìm sự ủng hộ cho luận điểm của mình.
Lợi dụng quyền lực nặc danh. Trong trường hợp này, người ngụy biện không nêu danh tính người có thẩm quyền, và vì không ai biết tên người có thẩm quyền nên không ai có thể kiểm chứng sự chính xác của lời phát biểu. Một loại ngụy biện khác có quan hệ với loại này là dùng lời đồn đại để làm cơ sở lập luận.
Lợi dụng tác phong. Loại ngụy biện này dùng tác phong hay cách làm việc hay một đặc tính nào đó của đối tượng để cố thuyết phục về sự hợp lí của phát biểu.
Luận điệu cá trích. Loại ngụy biện này thường hay được ứng dụng khi một người nào đó đưa vào những phát biểu không dính dáng gì đến vấn đề đang tranh luận, nhằm mục đích đánh lạc hướng vấn đề.
Luận điệu ngược ngạo (Burden of Proof). Bằng chứng luôn luôn là gánh nặng của người phát biểu. Do đó, tìm cách di chuyển gánh nặng đó cho một người khác là một thủ thuật giới ngụy biện hay dùng. Ví dụ: “Anh không tin là có UFO, vậy anh có thể chứng minh không?”, đáng lẽ người phát biểu phải chứng minh là có UFO, nhưng công việc đó đã được chuyển cho người đối thoại.

Lợi dụng cảm tính và đám đông

Dựa vào bạo lực (ad baculum). Ngụy biện dựa vào bạo lực thực chất là một sự đe dọa, nhằm mục đích gây áp lực cho người đối thoại phải chấp nhận một kết luận nào đó. Ví dụ: “Không đồng ý với tôi thì vào game sẽ bị kill red”
Lợi dụng lòng thương hại (ad misericordiam). Đây là một loại ngụy biện dựa vào lòng trắc ẩn của người đối thoại để người đối thoại chấp nhận lí lẽ của mình. “Tôi hi vọng anh sẽ chấp nhận đề nghị này, chúng ta đã tiêu ra ba tháng nay rồi đấy.”
Lợi dụng hậu quả (ad consequentiam). Ngụy biện loại này thường được biểu hiện qua cách phát biểu “A hàm ý B, B là sự thật, do đó A là sự thật”.
Lạm dụng chữ nghĩa. Đây là một loại ngụy biện dựa vào dùng những chữ mang cảm tính cao để gắn một giá trị đạo đức vào một đề nghị hay một câu phát biểu.
Dựa vào quần chúng (ad numerum). Loại ngụy biện này tin rằng nếu có nhiều người ủng hộ một đề nghị nào đó, thì đề nghị đó phải đúng. Ví dụ như “Đại đa số thành viên cũ của [Clan Spammer TM] ủng hộ Phantom, hẳn anh ấy phải luôn luôn đúng.” Liên hệ với loại ngụy biện này là hình thức tranh thủ sự ủng hộ của đám đông để cố gắng cho thấy luận điểm của mình là đúng.

Làm lạc hướng vấn đề

Lí lẽ chẻ đôi. Loại ngụy biện này thường phân định một vấn đề thành hai giá trị: trắng và đen, bạn và thù, có và không, v.v.. dù trong thực tế, có hơn hai lựa chọn. Ví dụ nói: “Bạn chọn đi, hoặc bán cây Peril cho tui giá 10m adena, hoặc đưa tui 5m adena ngay lập tức!”
Lí lẽ ngờ nghệch (ad ignorantiam). Loại ngụy biện này, như tên gọi ám chỉ, xuất phát từ sự ngờ nghệch. Một trong những cách nói thông thường nhất trong loại ngụy biện này mà giới ngụy biện dùng là nếu một điều gì đó chưa được chứng minh là sai (hay giả) thì điều đó là đúng (hay thật).
Lí luận lươn trạch. Loại ngụy biện này cho rằng nếu một sự kiện xảy ra, các sự kiện có hại khác sẽ xảy ra.
Mệnh đề rời rạc. Đây là loại ngụy biện dùng hai (hay nhiều hơn hai) mệnh đề chẳng dính dáng gì với nhau để làm thành một phát biểu hay kết luận. Ví dụ “Mày ngưng lừa đảo giá bán SS chưa”. Câu hỏi thực ra hỏi đến 2 vấn đề “Mày lừa đảo giá bán SS” và “Mày đã ngưng việc lừa đảo đó chưa” trong khi sự thật là người được hỏi chưa hề lừa đảo.
Đơn giản hóa. Đây là một loại ngụy biện mà người phát biểu cố tình biến một quan niệm trừu tượng thành một điều cụ thể để bắt lấy thế thượng phong trong đối thoại (nhưng là ngụy biện). Ví dụ: “Tôi để ý thấy anh mô tả Phantom là một người láo xược. Vậy tôi hỏi anh cái “láo xược” đó nó nằm ở đâu trong bộ não? Anh không chỉ ra được cho tôi; do đó, tôi có thể nói cái láo xược không có thực.”

Qui nạp sai

Khái quát hóa vội vã. Loại ngụy biện này cũng khá phổ biến. Nó dùng một ví dụ hay trường hợp nhỏ và từ đó khái quát hóa cho một cộng đồng. Chẳng hạn như “PhuongDoan nhìn rất íu đúi, bởi vậy tụi [Clan Spammer TM] cũng íu đúi rốt”
Khái quát hóa không đúng chỗ. Đây là loại ngụy biện mà người sử dụng chúng thường áp dụng một qui luật chung cho một tình huống hay một cá nhân. Chẳng hạn như “Clan FMW không ngại kill red người khác. Bé LeQuyen1987 cũng thuộc FMW, vậy bé ấy hẳn là thường xuyên kill red người khác.”
Kéo dài tính tương đồng. Trong loại ngụy biện này, người dùng nó đề nghị một điều lệ chung chung, rồi ứng dụng nó cho mọi trường hợp và cá nhân. Ví dụ: “Tôi tin rằng chống luật cấm spam bằng cách cố ý spam là một điều sai trái”
Lí lẽ quanh co. Loại ngụy biện này thường lẩn quẩn trong vài giả định và kết luận. Chẳng hạn như “Prophet nhất định không thể làm clan leader được. Do đó không thể để Prophet làm clan leader. Ví thể, các Prophet sẽ không cần phải đi đăng ký đánh thành. Do vậy, Prophet không thể nào làm leader được.” Tức là trong một lí giải như thế, cả hai giả thuyết và kết luận đều giống nhau.
Đảo ngược điều kiện. Loại ngụy biện này thường được biểu hiện qua hình thức “Nếu A xảy ra thì B sẽ xảy ra, do đó, nếu B xảy ra thì A sẽ xảy ra.”
Lợi dụng rủi ro. Ngụy biện này thường dùng một qui luật chung và áp dụng nó cho một trường hợp cá biệt. Ví dụ: “Luật giao thông không cho anh chạy quá 50 km/h. Cho dù cha anh sắp chết anh cũng không được chạy quá tốc độ đó.”
Lợi dụng trường hợp cá biệt. Ngụy biện này thường dùng một trường hợp cá biệt để đem ra ứng dụng cho một đám đông. Ví dụ: “A bận ôn thi nên không để đi kiếm tiền để nộp quỹ clan, vì thế A được miễn nộp quỹ clan tuần này. Mà thế là không công bằng, nên leader phải miễn quỹ clan tuần này cho tất cả thành viên.”
Kết luận lạc đề. Loại ngụy biện này thường xuất hiện khi một kết luận chẳng dính dáng gì đến lí lẽ mà người biện luận trình bày. Một ví dụ tiêu biểu cho trường hợp ngụy biện này là: “Độ nhiễm sắt trong nước ở Việt Nam chưa cao và còn trong mức độ cho phép. Dữ kiện của Bangladesh cho thấy tình trạng nhiễm sắt ở Việt Nam rất trầm trọng.”
Ngụy biện rơm. Loại ngụy biện này cố tình xuyên tạc, bóp méo quan điểm hay phát biểu của người khác, để làm luận điểm tấn công. Đây là một ngụy biện, vì nó không đương đầu với cái lí lẽ đang bàn. Chẳng hạn như: “Khi không có party thì mọi người nên đi solo vào lair. Nhiều người từ chối solo vào lair, nhưng họ nên thấy rằng vào lair train hiệu quả hơn là lê lết buôn dưa ở Giran." Trong đây, ý tranh luận của người trước là không nên solo vào lair nhưng người phản biện lại nói đến tính hiệu quả khi train ở lair - điều mà ai cũng biết.

Nguyên nhân giả

“Postology”. Loại ngụy biện này phát biểu rằng hai sự kiện xảy ra, một trước và một sau, có quan hệ với nhau như nguyên nhân và hậu quả. Ví dụ: “Vài nick bị ban khi spam, vậy chúng ta không nên spam.” Thực tế là bạn có thể spam mà vẫn không bị ban nếu spam đúng chỗ.
Ảnh hưởng liên đới. Một sự kiện được cho là có ảnh hưởng đến một sự kiện khác, nhưng thực chất thì cả hai sự kiện đều có cùng một nguyên nhân. Đây cũng chính là một trường hợp ngụy biện dưới dạng “postology”. Ví dụ: “Chúng ta đang chứng kiến tình trạng Summoner không kiếm được clan rất cao, vì không clan nào muốn chứa Summoner” (Nhưng có thể cả hai sự kiện có nguyên nhân là một summoner cụ thể nào đó hay chơi bẩn nên không clan nào nhận.)
Ảnh hưởng không đáng kể. Đây là một loại ngụy biện mang tính phóng đại từ một ảnh hưởng rất nhỏ. Chẳng hạn như “Hút thuốc gây ra ô nhiễm môi trường” là một phát biểu đúng, nhưng ảnh hưởng của thuốc lá đến môi trường rất khiêm tốn khi so với ảnh hưởng của khói xe, chất thải công nghiệp từ các nhà máy
Ảnh hưởng ngược chiều. Mối quan hệ giữa nguyên nhân và hậu quả bị đảo ngược chiều để tìm đến một kết luận mang tính ngụy biện. Ví dụ: “Ung thư gây ra thói quen hút thuốc lá”.
Nguyên nhân phức tạp. Một sự kiện xảy ra có thể do nhiều nguyên nhân khác nhau, nhưng người ngụy biện có thể đơn giản hóa thành một liên hệ đơn giản. Chẳng hạn như “Tai nạn xe cộ là do đường xá xấu” có thể đúng, nhưng tai nạn cũng có thể do người lái xe ẩu trong một điều kiện xấu.
Nguyên nhân sai (Non causa pro causa). Loại ngụy biện này xảy ra khi một điều nào đó được cho là nguyên nhân của một sự kiện, nhưng nó chưa thực sự được chứng minh là nguyên nhân. Ví dụ: “Tôi uống một viên aspirin và nhớ đến Lineage, và tôi không còn bị nhức đầu. Như vậy Lineage đã chữa trị tôi khỏi nhức đầu.”

Nhập nhằng

Lí lẽ mơ hồ. Dùng những chữ và lí lẽ mơ hồ, tối nghĩa là một hình thức ngụy biện, nhất là khi một chữ hay câu phát biểu được dùng với hai (hay nhiều hơn hai) ý nghĩa khác nhau. Ví dụ: “Giết người là xấu, vì thế những người đi giết những người đang red cũng là xấu”
Chơi chữ (Amphiboly). Ngụy biện bằng chơi chữ dựa vào những giả thuyết mơ hồ, nhập nhằng, do bất cẩn thận hay cách phát biểu sai văn phạm.
Trọng âm (accent). Đây là một hình thức ngụy biện bằng cách dùng thay đổi ý nghĩa của một câu văn qua nhấn mạnh.

Phạm trù sai

Hỗn hợp. Loại ngụy biện này thường dùng những đặc tính bề ngoài để suy luận cho một điều gì cá biệt. Ví dụ: “Xe đạp được làm bằng những dụng cụ nhẹ kí, do đó, xe đạp rất nhẹ”, hay “Xe hơi dùng ít xăng dầu và không gây ra ô nhiễm môi trường bằng xe bus. Do đó, xe hơi không gây hại cho môi trường bằng tác hại của xe bus.”
Phi thể thức (ad hoc). Giải thích và lí lẽ là hai điều khác nhau. Nếu muốn xác minh A, và dùng B làm bằng cớ, thì câu phát biểu “A xảy ra bởi vì B xảy ra" là một lí lẽ. Tuy nhiên, nếu muốn xác minh một sự thật về B, thì câu phát biểu “A xảy ra bởi vì B xảy ra” không phải là một lí lẽ mà là một lời giải thích. Ngụy biện theo kiểu phi thể thức là hình thức dùng giải thích sau khi đã có sự thật mà sự thật không ứng dụng vào một bối cảnh khác. Thông thường ngụy biện phi thể thức được khoác vào chiếc áo lí lẽ. Chẳng hạn như nếu chúng ta giả định rằng clan leader đối xử công bằng với mọi member trong clan, thì những phát biểu sau đây là những lời giải thích phi thể thức: “Tôi mới được leader cho mượn tiền”, “Nói clan leader đi, ảnh nhiều tiền lắm”, “Nhưng ảnh có cho những người khác mượn tiền không”, “À, leader rất khó hiểu.”

Phi logic (non sequitur) và nhầm lẫn trong tam đoạn luận

Phi logic. Ngụy biện phi logic thường xảy ra trong trường hợp một lí lẽ mà kết luận được rút ra từ những tiêu đề không dính dáng gì với nhau. Chẳng hạn như “Người Ai Cập đã từng làm nhiều khai quật để xây dựng những kim tự tháp, họ chắc chắn phải rất thạo về cổ sinh vật học.”
Loại bỏ tiền đề. Ngụy biện loại này thường xảy ra dưới hình thức “nếu A thì B, không phải A thì không phải B.” Ví dụ: “Nếu tôi ở TPHCM thì tôi đang ở Vietnam. Tôi hiện không ở TPHCM, do đó, tôi không ở Vietnam”.
Giả định hư. Đây là một loại ngụy biện bằng cách dùng kỹ thuật phỏng vấn. Một trường hợp cổ điển là “Ông đã ngưng đánh vợ chưa?” Tức là một câu hỏi với một giả định rằng người được hỏi từng hành hung vợ.
Ngụy biện bốn ngữ. ( Một tiêu chuẩn của tam đoạn luận gồm có 3 chữ). Ví dụ như trong câu phát biểu “Tất cả chó là thú vật, và tất cả mèo là loài động vật có vú, do đó tất cả chó là loài động vật có vú,” có bốn chữ: chó, mèo, động vật, và động vật có vú.
Đứt đoạn. Hai sự vật riêng biệt được xem là có liên hệ nhau nếu chúng có chung đặc tính. Người ngụy biện lợi dụng chữ giữa của một phát biểu để đưa đến một kết luận sai.

Các nhầm lẫn khác

Dẫn chứng bằng giai thoại. Một trong những ngụy biện phổ biến nhất và đơn giản nhất là dựa vào những câu chuyện có tính vụn vặt, hay giai thoại.
Lợi dụng cổ tích. Đây là một loại ngụy biện cho rằng những gì đúng hay tốt chỉ đơn giản vì chúng là cổ xưa, và những người theo cách ngụy biện này thường nói “hồi nào đến giờ ai cũng vậy.”
Dựa vào cái mới (ad novitatem). Ngược lại với loại ngụy biện dựa vào cái cũ, ngụy biện dựa vào cái mới cho rằng một điều gì đó tốt hơn và đúng hơn đơn giản chỉ vì nó mới hơn cái khác. “Windows XP phải tốt hơn Windows 2000.”
Lí lẽ của đồng tiền. Loại ngụy biện này thường dựa vào một niềm tin duy nhất rằng đồng tiền là một tiêu chuẩn của sự đúng đắn. Những người có nhiều tiến có khả năng đúng hơn những người ít tiền.
Dựa vào cái nghèo. Ngược lại với ngụy biện dựa vào sự giàu có, có một loại ngụy biện khác dựa vào sự nghèo khổ.
Điệp khúc (ad nauseam). Loại ngụy biện này cho rằng một lí lẽ càng được lặp đi lặp lại nhiều chừng nào thì nó sẽ được người ta chấp nhận là đúng. Do đó, người ngụy biện thường chỉ lặp đi lặp lại những phát biểu, bất kể là quái dở thế nào, cho đến khi người đối thoại mệt mỏi không còn muốn nghe nữa.
Lạm dụng thiên nhiên. Tìm cái tương đồng giữa một kết luận nào đó và một khía cạnh của thế giới tự nhiên, rồi từ đó phát biểu rằng kết luận đó là không thể tránh khỏi.
Ngụy biện “Tu quoque”. Đây là một trong những ngụy biện rất phổ biến. Nó dựa vào lí lẽ rằng một hành động có thể chấp nhận được bởi vì người đối nghịch đã làm. Chẳng hạn như “Anh là một người lừa dối.” “Rồi sao? Anh cũng là một tay lừa dối vậy.”
Lạm dụng thống kê. Thống kê thường được giới ngụy biện sử dụng tối đa, vì theo họ thống kê có thể dùng để “chứng minh” bất cứ điều gì. Người ta có thể vặn vẹo hai con số 1 và 3 điểm để sản xuất những phát biểu như “khác nhau 2 điểm”, “cao gấp 3 lần”, hay “tăng 200%”; người ta có thể dựa vào ý kiến đồng tình của 4 người trong 5 người để cho là “80% người được thăm dò”, hay thậm chí “đa số cộng đồng” đồng ý với một luận điểm nào đó. Tức là những khái quát hoá một cách vội vã, hay dựa vào một mẫu số cực kỳ thấp, thấp đến độ nó không có nghĩa lí gì. Thực ra, thống kê không chứng minh điều gì cả. Thống kê chỉ là một phương tiện hay thuật toán dùng để loại bỏ những trường hợp khả dĩ hay không khả dĩ. Vì có quá nhiều ngụy biện thống kê, nên vấn đề này sẽ được bàn tiếp trong một dịp khác. Tuy nhiên, những ai thích tìm hiểu vấn đề ngụy biện thống kê có thể tìm đọc cuốn sách rất nổi tiếng của Darrell. Huff, có tựa đề là “How to lie with statistics” (tạm dịch: “Làm thế nào để lừa dối bằng thống kê”).

Mặc định Ðề: Các hình thức ngụy biện khi tranh luận

Có thể nói những loại ngụy biện trên đây có những đặc điểm chung là (a) phát biểu không dựa vào lí lẽ logic; (b) các định đề không vững để đi đến một kết luận; và (c) đưa ra giả định không đúng. Ngụy biện, do đó, nói cho cùng, là một sản phẩm của sự lười suy nghĩ. Và hầu như trong chúng ta, ai cũng có ít nhất là một lần lười suy nghĩ. Do đó, nếu điểm qua những loại ngụy biện trên đây, chúng ta tự cảm nhận rằng trong quá khứ mình chắc cũng có lần phạm vào lỗi lầm của ngụy biện. Điều này có thể đúng, và không nên lấy làm ngạc nhiên.

Nhưng tại sao những ngụy biện vẫn còn có mặt? Theo tôi, bởi vì chúng vẫn có khách hàng. Vẫn có người, dù ít hay nhiều, tin tưởng vào ngụy biện, vì nó thuận nhĩ, trơn tru, và nhất là không thách thức. Sờ một hòn đá trơn tru đem lại cho chúng ta một cảm giác khoan khoái dễ chịu hơn là sờ một hòn đá lởm chởm, hay ngồi trên một cái ghế ghồ ghề. Người ta thích sự trơn tru, bởi vì trơn tru là dấu hiệu của sự khoan khoái, dễ chịu, là cái khoảng thời gian giải lao, không cần sự thách thức.

Có lẽ, ở một khía cạnh nào đó, điều này cũng không đến nỗi tệ, bởi vì những ngụy biện phản ánh sự thành công [hay có người nói sự phong phú] của ngôn ngữ trong việc tách rời giữa những gì thô thiển, gồ ghề với những gì hoàn thiện, mỹ miều. Nhưng sự trơn tru của các vật thể và ngôn ngữ ngày nay đem lại cho chúng ta một cảm giác giả tạo về thế giới thực của các vật thể. Những kỳ kẹt xe trên đường xá mới để lộ trái tim phức tạp của một thành phố. Tương tự, một sự cố của internet sẽ nhắc nhở chúng ta về tình trạng hỗn mang và phức tạp của hệ thống thông tin điện tử. Sự hỗn mang và phức tạp là thực. Trơn tru, tròn trĩnh có thể là giả tạo. Những câu văn ngụy biện có thể chỉ là những lời phát biểu lém lỉnh thay vì lịch thiệp, hàm chứa mánh khóe thay vì thân thiện. Có thể nói, ngụy biện là những lối sáo ngữ liến thoắng nhằm vào mục đích lôi cuốn người nghe/đọc, thay vì cung cấp cho họ một sự thực.

Bởi vì ngụy biện là những lí lẽ mà bề ngoài có vẻ logic, nên chúng có khả năng thuyết phục những người không chịu khó suy nghĩ, nhất là những người còn mang nặng cảm tính. Điều này giải thích tại sao nhiều người trong chúng ta tiếp nhận một cách thụ động quá nhiều những điều càn rỡ về thế giới chung quanh, kể cả những niềm tin tôn giáo, những mê tín dị đoan, những triết lí quái đảng, những thông tin sai lạc, v.v.. Cái tác hại của việc tiếp nhận thụ động này là nó làm cho chúng ta trở nên nô lệ với cảm tính, và dễ dàng trở thành những người cuồng tín.

Để không trở thành những nô lệ, chúng ta cần phải suy nghĩ nghiêm túc. Suy nghĩ nghiêm túc là một quá trình hoạt động tri thức nhằm ý niệm hóa, ứng dụng, phân tích, tổng hợp, và (hay) đánh giá những thông tin được thu thập từ quan sát, kinh nghiệm, phản ánh, lí luận, hay liên lạc, như là một niềm tin cho hành động. Chúng ta cần phải dựa vào những giá trị tri thức với những đặc điểm như trong sáng, chính xác, nhất quán, có liên hệ, bằng chứng tốt, lí lẽ hợp lí, có chiều sâu, và công bình. Tức là, trước một câu phát biểu hay một đề nghị, chúng ta phải thẩm định lại kết cấu và nguyên tố của phát biểu hay đề nghị đó. Những kết cấu và nguyên tố này là: mục đích, vấn đề, giả định, quan niệm, bối cảnh, kết luận, ngụ ý, hậu quả, phạm vi tham khảo, và quan điểm khác.

Theo dõi những tranh luận trên các forum, chúng ta thấy những hình thức tấn công cá nhân (thay vì tấn công vào luận điểm), xuyên tạc ý tưởng, chụp mũ, suy luận theo cảm tính, mỉa mai, đơn giản hóa vấn đề, v.v… xuất hiện hầu như hàng ngày, có khi hàng giờ. Vì những tần số của những loại ngụy biện xuất hiện quá nhiều như thế, nó thành một sự rập khuôn. Theo thời gian, rập khuôn trở thành “truyền thống”. Hậu quả của cái truyền thống này là những ai ra ngoài cái khuôn sáo của ngụy biện đều có thể bị xem là phi chính thống, dẫn đến một lối suy nghĩ và phán xét kỳ quặc

Trong cái sự thực phức tạp, mờ mờ ảo ảo của vấn đề, có cái đẹp riêng. Không phải cái đẹp trơn tru, tròn trĩnh, nhưng là cái đẹp khắt khe của sự thật. Tương tự, một lời phát biểu nghịch lý có cái đẹp của nó, vì nó có thể đánh thức chúng ta về một thế giới phức tạp, một thế giới không nằm gọn trong đúng/sai, tốt/xấu, bạn/thù. Có lẽ đã đến lúc chúng ta nên vượt qua chính mình.

Apache httpd 2.4 and PHP 7 in macOS

Mon, 25 Sep 2017 00:00:00 +0000

Together, Apache HTTP server, PHP, and MySQL form a powerful and popular combination for Web development. MacOS are often shipped with pre-installed versions of Apache HTTP server and PHP but these are often outdated and merely customised for macOS. The well-known bundles *AMP (e.g. WAMP for Windows, LAMP for Linux, MAMP for Mac) are commonly used but also considered a tad bloated for the beginners like me ;).

After few years of Java development, I turned myself into Web development, and in particular, PHP programming. As usual, I would start with pure PHP aspects that really help me to understand the fundamental concepts and techniques instead of being drown with everyone-known frameworks and their complexity and hard-to-understand magics. As a result, I first looked for a simple setup of Apache and PHP that best suits the beginning of my initial learning path. In this note, I write down what I learn from many sources on the Internet tweaked to suit my needs.

Goal

Let’s assume that we want to set up an exemplary Web development environment including Apache HTTP 2.4 server and PHP 7.

Strategies

A typical and well-documented approach is to load PHP processor as a module under Apache httpd using the directive LoadModule. It is so-called mod_php approach. However, this is now gradually out of favor of Web developers and hosting providers because the tight combination of PHP and Apache makes things difficult for monitoring, debugging, and scaling. One of the recent favorite strategies is to set up PHP as Fast-CGI using PHP-FPM (FastCGI Process Manager). This method also brings several advantages including good support for nginx integration and performance.

Based on the aforementioned analysis of strategies, I consider the following installation activities to fulfill that goal.

Installing Homebrew (a sane package manager for macOS)
Installing Apache HTTP server 2.4
Installing PHP 7 and PHP-FPM
Putting all together
Installing DNSMasq (optional)

Installation and Configuration

Homebrew

Homebrew is currently a prominent and well-supported package managing solution for macOS. Installing Homebrew is rather straightforward. Nevertheless, you should be familiar with using the command line in order to easily get thing done. All you need is to launch the Terminal app (/Applications/Utilities/Terminal.app) or even the better iTerm 2 and start executing terminal commands.

Note: From now on, the sign $ (if exists) will denote the user’s command line prompt where you will execute the terminal commands, except that you must not type $.

Homebrew requires Xcode’s command line tooling so we must install it first.

# Install Xcode command line tooling
$ xcode-select --install

Next we start installing Homebrew using a one-line command.

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

By default, Homebrew will be installed into /usr/local/Cellar and use /usr/local and its sub-folders for the its installed binary and configuration files. For more advanced settings and manual configurations, please refer here. To ensure that Homebrew is properly installed, we can simply check.

$ brew --version
Homebrew 1.3.8
Homebrew/homebrew-core (git revision 22ac3; last commit 2017-12-02)

The command brew config shows more information about the installed Homebrew.

brew config

Now we should add some extra taps (i.e. package repositories in Homebrew’s world) that contain packages we need, for instance, Apache HTTP server, PHP, and MySQL along with tools for running them as macOS services.

brew tap homebrew/apache
brew tap homebrew/php
brew tap homebrew/services

Last but not least, always make sure Homebrew is up-to-date.

brew update
brew doctor

Apache HTTP server

Installing Apache HTTP server 2.4

We can also use the pre-installed Apache HTTP server (from now on, Apache httpd, Apache, httpd will be used interchangeably) shipped with macOS. Nevertheless, I want to play around with the newer version. Apart from the Homebrew-based installation commands, the steps for setting up Apache are the same in either way.

brew install httpd

NOTE 1: Since Sep 30th, 2017 the Apache HTTP server package in Homebrew has been renamed from httpd24 to httpd and the corresponding folders are also changed from apache2 to httpd. This note has been updated with the newer versions. Apart from the aforementioned updates, the configuration steps remain the same though.

NOTE 2: When we use brew install <package_name without any further options, Homebrew often accomplishes the installation faster by downloading a bottled version (i.e. a pre-compiled package) from homebrew.bintray.com. If there are any extra compling options, Homebrew can download the package’s source code and compile the source. The compilation of a source package usually takes a a bit longer. For a simple and quick start, I mostly opt for the bottled versions of Apache server and PHP as the configured and compiled options are rather sufficient. After learning the fundamental aspects, I can turn to a more complex approach with lots of tweaks for further needs or experiments.

By default, the Homebrew-based Apache server will use the ports http/8080 and https/8443 that do not need system administrator privileges. In my setup, I will go with this option as I simply want to run Apache with my user account. Nonetheless, you can tell Homebrew to install a version that use ports 80 and 443 by the option --with-privileged-ports.

brew install httpd --with-privileged-ports

In case you want to experiment with the bleeding edge Apache server pulled from its development repository, use the option --HEAD along with other options. For example,

brew install httpd --HEAD --with-privileged-ports

Let’s assume the bottled httpd is installed. The Apache server will be installed in the following folder /usr/local/Cellar/httpd/2.4.29 (which might be different in your computer). You can check the Homebrew’s Cellar, i.e. where Homebrew puts installed packages, with brew --cellar. The precise location of httpd in a bit complex nerdy form is $(brew --cellar)/httpd/$(brew list --versions httpd | cut -d ' ' -f 2).

cd $(brew --cellar)/httpd/$(brew list --versions httpd | cut -d ' ' -f 2)

Configuring Apache server

The main configuration file httpd.conf can be located at /usr/local/etc/httpd. There are some directives and options that you may want to notice or change ( # starts a comment).

There is a default directive Listen 8080 denoting the port where Apache server will be listening. You can change it to your favourite one or the one that suits your projects. Note that if you change to a port in the range from 1-1023 (so-called privileged ports), you need an administrator role to run the httpd process. Because I need to create virtual hosts for various projects, I have to configure httpd to listen at a particular address and port, which is 127.0.0.1:8080 in my setting.

The directives LoadModule will enable some Apache HTTP modules we need for configuring PHP-FPM such as mod_proxyand mod_proxy_fcgi (disabled by default).

Listen 127.0.0.1:8080
...
# LoadModule proxy_module libexec/mod_proxy.so
...
# LoadModule proxy_fcgi_module libexec/mod_proxy_fcgi.so
...

Apache server can be started by using either apachectl start or brew services start httpd. Note that the latter also installs a snippet to run Apache httpd as a startup service.

brew services start httpd
brew services list

Running and Testing

We use the following commands to start, stop, or restart httpd, respectively.

brew services start httpd
brew services stop httpd
brew services restart httpd

To see whether Apache httpd is up and running, you just start your Web browser and point to the URL http://localhost:8080 and should see a simple Web page saying “It works”. We can also check from the command line as well.

$ launchctl list | grep httpd
91962	0	homebrew.mxcl.httpd

$ ps -ef | grep httpd
  501 91962     1   0  8:51AM ??         0:00.07 /usr/local/opt/httpd/bin/httpd -D FOREGROUND
  501 91968 91962   0  8:51AM ??         0:00.04 /usr/local/opt/httpd/bin/httpd -D FOREGROUND
  501 91969 91962   0  8:51AM ??         0:00.04 /usr/local/opt/httpd/bin/httpd -D FOREGROUND
  501 91970 91962   0  8:51AM ??         0:00.04 /usr/local/opt/httpd/bin/httpd -D FOREGROUND
  
$ lsof -Pni4 | grep httpd
httpd     91962 huytran    5u  IPv4 0xd51255ae3385c3d7      0t0  TCP *:* (CLOSED)
httpd     91968 huytran    5u  IPv4 0xd51255ae3385c3d7      0t0  TCP *:* (CLOSED)
httpd     91969 huytran    5u  IPv4 0xd51255ae3385c3d7      0t0  TCP *:* (CLOSED)
httpd     91970 huytran    5u  IPv4 0xd51255ae3385c3d7      0t0  TCP *:* (CLOSED)

We can also examine whether Apache server configuration file is correct:

$ apachectl configtest
Syntax OK
$ httpd -t
Syntax OK

PHP 7

Install PHP and PHP-FPM

Likewise, I just use Homebrew to install newer/older versions of PHP ranging from 5.x to 7.x. I opt for the stable release of PHP 7.

brew install php72

The default installation of PHP 7 almost covers several useful aspects including a PHP module libphp7.so that can be integrated with Apache HTTP server via the directive LoadModule and PHP-FPM that can be used as FastCGI.

$ php -version
PHP 7.2.0 (cli) (built: Dec  2 2017 11:27:08) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2017 Zend Technologies

PHP configurations

The main configuration file of PHP is php.ini as shown below.

Configuration File (php.ini) Path: /usr/local/etc/php/7.2
Loaded Configuration File:         /usr/local/etc/php/7.2/php.ini
Scan for additional .ini files in: /usr/local/etc/php/7.2/conf.d
Additional .ini files parsed:      (none)

The default setting in /usr/local/etc/php/7.2/php.ini is quite sufficient for simple development purposes. We may tweak later if necessary but can leave it for now.

Starting PHP-FPM

The configuration for PHP-FPM is /usr/local/etc/php/7.2/php-fpm.conf. Note the last lines in this file

;include=/usr/local/etc/php/7.1/php-fpm.d/*.conf

You can remove the semi-colon to enable the inclusion of other configurations and/or might also change the path to whether it fits your development environment. In the folder /usr/local/etc/php/7.1/php-fpm.d/*, there is a file www.conf that can be used as a starting point for setting up your own PHP-FPM.

Now we can start PHP-FPM to see whether everything is fine before doing some extra configuration steps.

$ brew services start php72
==> Successfully started `php72` (label: homebrew.mxcl.php72)
$ brew services stop php72
Stopping `php72`... (might take a while)
==> Successfully stopped `php72` (label: homebrew.mxcl.php72)
$ brew services restart php72

The command brew services start php72 can create the file homebrew.mxcl.php72.plist in ~/Library/LaunchAgents for starting up.

$ launchctl list | grep php
47728	0	homebrew.mxcl.php72
$ ps -ef | grep php-fpm
...

Putting all together

After basically finishing the installation of Apache, PHP 7 and PHP-FPM, we start putting these pieces together.

Configurating a ‘PHP-FPM’ pool

PHP-FPM supports multiple resource pools. Each pool defines how PHP-FPM will create and manage processes. Let’s start with enabling the inclusion and handling of pools in /usr/local/etc/php/7.2/php-fpm.conf by changing the following line

;include=/usr/local/etc/php/7.2/php-fpm.d/*.conf

to (i.e. uncommenting it)

include=/usr/local/etc/php/7.2/php-fpm.d/*.conf

Then we can leverage the stock configuration in /usr/local/etc/php/7.2/php-fpm.d/www.conf. In my case, I have cleaned up www.conf and kept a simple configuration as following for the sake of readability. The configuration explains for itself.

[www]
user = _www
group = _www
listen = 127.0.0.1:9072
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

The most important setting of each pool is the TCP socket (including IP address and port) or Unix domain socket (UDS) that PHP-FPM will be receiving FastCGI requests. This will be set using the directive listen . A typical setting of PHP-FPM is listen = 127.0.0.1:9000. As I might want to install multiple PHP versions for testing, I will change the port to 9072 that corresponds to the PHP version 7.2.

Note that we can also configure PHP-FPM to serve at a Unix domain socket (UDS) because PHP-FPM and httpd processes are running in the same host. To do that, the directive listen must be changed, for example, listen = /usr/local/var/run/php72-fpm.sock. This approach would need extra effort to configure Apache to use PHP-FPM via UDS that we will visit later near the end of this guide.

Now as I want to check whether PHP-FPM will be up and running with the above changes, I restart PHP-FPM and check the open ports 9072.

$ brew services restart php72
$ ps -ef | grep php-fpm
...
$ lsof -Pni4 | grep php-fpm
php-fpm   53180 huytran    6u  IPv4 0xd51255ae2d4097b7      0t0  TCP 127.0.0.1:9072 (LISTEN)
php-fpm   53187 huytran    0u  IPv4 0xd51255ae2d4097b7      0t0  TCP 127.0.0.1:9072 (LISTEN)
php-fpm   53188 huytran    0u  IPv4 0xd51255ae2d4097b7      0t0  TCP 127.0.0.1:9072 (LISTEN)

PHP-FPM Delegation

I can define virtual hosts in Apache that suit my needs for separating different development projects. Instead of creating one large httpd.conf, I will create separate virtual host configurations. Again, I will start with the exemplary virtual host configuration provided in /usr/local/etc/httpd/extra/httpd-vhosts.conf.

Enabling ‘mod_proxy’ and ‘mod_proxy_fcgi’

Change the following lines in /usr/local/etc/httpd/httpd.conf

# LoadModule proxy_module libexec/mod_proxy.so
...
# LoadModule proxy_fcgi_module libexec/mod_proxy_fcgi.so

LoadModule proxy_module libexec/mod_proxy.so
...
LoadModule proxy_fcgi_module libexec/mod_proxy_fcgi.so

Defining Virtual Hosts

I want to define a PHP development project residing in /Users/huytran/working/dev/dev-web/php. Therefore, I change /usr/local/etc/httpd/httpd.conf

#Include /usr/local/etc/httpd/extra/httpd-vhosts.conf

Include /usr/local/etc/httpd/extra/httpd-vhosts.conf

and then create a file /usr/local/etc/httpd/extra/httpd-vhosts.conf

<VirtualHost *:8080>
    ServerName php7.test
    DocumentRoot "/Users/huytran/working/dev/dev-web/php"
    ProxyPassMatch ^/(.*\.php(/.*)?)$ fcgi://127.0.0.1:9072/Users/huytran/working/dev/dev-web/php/$1
    DirectoryIndex index.php index.html
	<Directory "/Users/huytran/working/dev/dev-web/php">
		Require all granted
	</Directory>
</VirtualHost>

Further information on Apache virtual hosts can be found here or here. I can briefly explain some relevant directives used in my virtual host configuration:

<VirtualHost>
- is used to define a new virtual host with respect to a specific hostname or IP address.
- Syntax: <VirtualHost addr[:port] [addr[:port]] ...> ... </VirtualHost>
  - Here I use the wildcard *:* to match any IP addresses and ports. Note that the port specified in a <VirtualHost> does not affect the real port Apache httpd is listening.
ServerName
- ServerName is recommended in each <VirtualHost> for resolution and matching. If absent, Apache will use the global ServerName stated in httpd.conf.
- In my case, I assign different ServerName, php7.test and htmljs.test, respectively, for each project to distinguish them. As Apache mainly matches virtual hosts via IP addresses and it will resolve host names based on DNS servers, I will configure corresponding hostname resolution for php7.test and htmljs.test using /etc/hosts or DNSMasq.
DocumentRoot
- DocumentRoot specifies the place where Apache httpd will look for files to serve relevant incoming request. In each case, DocumentRoot will be the absolute path of my development project.
ProxyPassMatch
- ProxyPassMatch is part of mod_proxy that can map remote servers into local URLs using regular expressions.
  - Syntax: ProxyPassMatch [regex] !|url [key=value [key=value ...]]
  - When Apache httpd receives a request for a certain PHP file (e.g. test.php) then it needs PHP to handle that request. The regular expression ^/(.*\.php(/.*)?)$ is used to check whether the incoming request is for a .php file.
  - $1 is called back-reference, as it refers to the matched part of the regular expression corresponding to the outermost pair of parentheses. In case the expression matches the incoming request URI (i.e. the request corresponds to a file .php, for instance, /test.php) $1 will represent the request URI. As a request is often treated as relative path from the DocumentRoot, we must add DocumentRoot path before $1 to form an exact absolute path and hand it over to PHP-FPM by adding the prefix fcgi://127.0.0.1:9072. Note that PHP-FPM is serving at 127.0.0.1:9072 and fcgi:// denotes the scheme FastCGI provided by mod_proxy_fcgi.
- The project htmljs.test does not need PHP processing, therefore I do not set the directive ProxyPassMatch there.
DirectoryIndex
- DirectoryIndex sets the list of resources for an indexing request. As I want to use PHP, I set index.php as index resource backed up by the default index.html. Note that DirectoryIndex can also be set globally in httpd.conf that affects the main server and can be inherited in all virtual hosts.
```
<IfModule dir_module>
    DirectoryIndex index.php index.html
</IfModule>
```
- Note that DirectoryIndex index.html is the default global option defined in httpd.conf.
<Directory>
- encloses the settings for folders, subfolders, and their contents.
  - Require all granted : access is allowed unconditionally (authorization mod_authz_core). We need this, otherwise Apache will return an error 403 Forbidden.

Hostname Resolution

As mentioned before, I will use the virtual host php7.test in my macOS to refer to my corresponding Web development projects. The hostname is not known by Apache. Thus, I must tell it how to resolve this hostname. In my setting, all virtual hosts are sharing in the same local computer with the address 127.0.0.1. A simple solution is to change the file /etc/hosts where macOS will look for when resolving hostnames. This solution requires sufficient administration privilege.

$ sudo open -e /etc/hosts
Password: (enter your password here)

After entering the administrator password, I will add two following lines to /etc/hosts.

127.0.0.1    php7.test

In case either you do not want to mess up system files like /etc/hosts or you have several development projects, you can leverage dnsmasq for automatically resolving hostnames with very simple configurations. Note that this is also the reason I chose the .test for all of my development projects such that I can use dnsmasq and only one line directive to resolve the TLD (Top-Level Domain) test. You can surely pick any TLD other than test .

Testing Apache and PHP

Apart from the aforementioned tests, we need to check whether the integration of Apache and PHP-FPM is working. Let’s create a file index.php in /Users/huytran/working/dev/dev-web/php

<?php
  echo 'PHP Development Project';

Restart httpd and PHP-FPM to apply these changes.

$ brew services restart httpd
$ brew services restart php72

Given no problems, the servers should be restarted smoothly. Then open your Web browser to the following URLs http://php7.test:8080. If you see the content created above, our configuration is working.

DNSMasq

Instead of altering system hosts /etc/hosts, I can make use of dnsmasq to conveniently resolve many hostnames used in my virtual hosts.

Installing DNSMasq

# update homebrew and install dnsmasq
$ brew update && brew install dnsmasq
# Copy the dnsmasq daemon configuration file to be loaded at system startup time
$ sudo cp $(brew list dnsmasq | grep /homebrew.mxcl.dnsmasq.plist$) /Library/LaunchDaemons/
# start dnsmasq
$ sudo launchctl load -w /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist

Configuring DNSMasq

After installing DNSMasq, let’s open the configuration file

$ open -e /usr/local/etc/dnsmasq.conf

and add the following line at the end.

address=/app/127.0.0.1

The above configuration line tells DNSMasq to resolve the TLD dev to the local IP address 127.0.0.1. You can now use any hostnames like this.is.a.development.project.test in Apache virtual hosts.

Configuring macOS to use DNSMasq

We have to tell macOS that DNSMasq is now a nameserver running at the local host. The configuration file is /etc/resolv.conf specifies the nameservers for looking up hostnames. On the one hand, we should not mess up the system setting. Moreover, macOS might overwrite /etc/resolv.conf when there are changes in Network setting. An elegant solution is to create a file in the folder /etc/resolver/with the exact TLD managed by DNSMasq.

$ sudo mkdir -p /etc/resolver/
$ sudo sh -c 'echo "nameserver 127.0.0.1" > /etc/resolver/test'

The aforementioned commands needs administrator privileges and will create a file test with the content nameserver 127.0.0.1 in the folder /etc/resolver/.

Testing DNSMasq

Now we can delete the lines added before in /etc/hosts and start dnsmasq (as root) and test the resolving of hostnames.

# start dnsmasq
$ sudo brew services start dnsmasq

# ping the hostnames
$ ping -c 1 php7.test
PING php7.test (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.040 ms

# query hostname using dnsmasq running at 127.0.0.1
$ dig php7.test @127.0.0.1
...
;; ANSWER SECTION:
php7.test.		0	IN	A	127.0.0.1

# test arbitrary hostnames ending with .test
$ dig an.arbitrary.host.test @127.0.0.1

Using Proxy FastCGI via Handler

Instead of configuring ProxyPassMatch for each individual virtual host to handle .php files, we can also set up directives in Apache httpd.conf (global scope) or at the beginning of the virtual host configuration localhost.conf (virtual host scope) via handlers.

First comment out or delete the line ProxyPassMatch ... in the virtual host configuration /usr/local/etc/httpd/extra/httpd-vhosts.conf.

# ProxyPassMatch ^/(.*\.php(/.*)?)$ fcgi://127.0.0.1:9072/Users/huytran/working/dev/dev-web/php/$1

Then, add the following lines to either the end of /usr/local/etc/httpd/httpd.conf or the beginning of /usr/local/etc/httpd/extra/httpd-vhosts.conf.

<Proxy "fcgi://localhost:9072/" enablereuse=on max=10>
</Proxy>
<FilesMatch "\.php$">
    <If "-f %{REQUEST_FILENAME}">
        SetHandler "proxy:fcgi://127.0.0.1:9072/"
    </If>
</FilesMatch>

<FilesMatch> : defines the scope of the enclosed directives by filenames that match the regular expression.
- Syntax: <FilesMatch regex> ... </FilesMatch>
SetHandler: forces matching files to be processed by a handler.
- Syntax: SetHandler handler-name|none|expression

Restart httpd

Handling PHP via Unix Domain Sockets

As mentioned before, PHP-FPM can serve on Unix domain sockets (UDS), too. UDS is widely used in the Unix/Linux world for inter-process communication. To enable PHP-FPM listening on UDS, open file /usr/local/etc/php/7.1/php-fpm.d/www.conf and change the directive listen .

[www]
...
listen = /usr/local/var/run/php72-fpm.sock
...

Now we must change the Apache configuration to adapt to UDS as well. In case we use ProxyPassMatch directive, change it like this:

ProxyPassMatch ^/(.*\.php(/.*)?)$ unix:/usr/local/var/run/php72-fpm.sock|fcgi://127.0.0.1:9072/Users/huytran/working/dev/dev-web/php

The change is to add unix:/usr/local/var/run/php72-fpm.sock| before fcgi://... and remove the captured request URI /$1 at the end.

Similarly, if we switch to Apache handler as above, we update the Apache configuration accordingly.

<Proxy "unix:/usr/local/var/run/php72-fpm.sock|fcgi://localhost:9072/" enablereuse=on max=10>
</Proxy>
<FilesMatch "\.php$">
    <If "-f %{REQUEST_FILENAME}">
        SetHandler "proxy:unix:/usr/local/var/run/php72-fpm.sock|fcgi://127.0.0.1:9072/"
    </If>
</FilesMatch>

We should ensure the configurations are good and then restart the servers afterwards.

# check Apache config
$ apachectl -t
Syntax OK
# check php-fpm config
$ php-fpm -t
[30-Sep-2017 19:39:46] NOTICE: configuration file /usr/local/etc/php/7.2/php-fpm.conf test is successful
# restart servers
$ brew services restart httpd
$ brew services restart php72

Cách xưng hô họ hàng (st)

Sun, 07 Aug 2016 00:00:00 +0000

Bài sưu tầm về cách xưng hô với họ hàng nội ngoại lưu lại đây thể tham khảo về sau.

Với họ nội (tức là họ hàng bên cha)

Các anh của ba và vợ của họ = bác
Các em trai của ba = chú.
Vợ của em trai của ba = thím.
Các em gái, chị gái của = cô.
Chồng của em gái, chị gái của ba = dượng.

Với họ ngoại (tức họ hàng bên mẹ)

Các anh, em trai của mẹ = cậu.
Vợ của anh trai, em trai của mẹ = mợ
Các chị, em gái của mẹ = dì.
Chồng các chị, em gái của mẹ = dượng.

Còn cách xưng hô bên gia đình sui gia

Vợ của em trai = em dâu.
Chồng của em gái = em rể.
Vợ của anh trai = chị dâu.
Chồng của chị gái = anh rể.

Đặc biệt bạn có tới 8 cố gồm 4 cố là ông bà nội ba, ông bà ngoại ba và 4 cố nữa là ông bà nội mẹ, ông bà ngoại mẹ.

Người miền Nam gộp chung các cố ông, cố bà (các cặp) thành một gọi cho dễ nhưng nên nhớ bạn chỉ có duy nhất 1 cố nội là ông bà nội của ba còn 3 cặp ông bà cố kia là cố ngoại hết theo quy định.

Khen con (st)

Wed, 01 Jul 2015 00:00:00 +0000

Thói quen khen ngợi con giỏi hay khả năng sẵn có của con

Mình để thấy không chỉ cha mẹ Việt mà cả cha mẹ người nước ngoài nữa thường hay con “giỏi quá” “thông minh quá” “sao con làm nhanh vậy”. Mới hôm qua thôi, mình để ý có một bà mẹ đố đứa con hơn 2 tuổi của mình “con chó kêu như thế nào”, đứa bé trả lời “gâu,gâu”, thế là bà mẹ khen luôn “giỏi quá”. Ngày trước có dịp đi nhà thờ, mình để ý thấy những người ở trong Nam, cứ trẻ con làm gì tốt hay ngoan cũng quen miệng khen “giỏi quá” “ngoan quá”.

Những khen ngợi như thế này có ảnh hưởng tới sự phát triển tính cách, cách nhìn nhận vấn đề của trẻ như thế nào?

Khen ngợi con giỏi, thông minh là không tốt

Khen ngợi con cái là một việc bố mẹ nên làm vì lời khen giúp trẻ được khích lệ và tự tin hơn. Nhưng không phải lời khen nào cũng có tác dụng như vậy. Nghiên cứu chỉ ra rằng, khen ngợi con cái là “giỏi và thông minh” một cách chung chung thậm chí sẽ có tác dụng ngược lại. Cách cha mẹ khen ngợi con cái cũng ảnh hưởng rất lớn đến khả năng đối mặt với thất bại, sẵn sàng chấp nhận thử thách và kiên trì trước những vấn đề khó.

Theo nghiên cứu của Carol Dwreck, Khi đối mặt với những vấn đề khó (hơn khả năng hiện tại của mình), những đứa trẻ quen được khen là thông minh có xu hướng rụt rè, không muốn giải quyết, do đó chúng tự hạn chế cơ hội học hỏi, nâng cao kiến thức, kĩ năng của mình. Ngược lại những đứa trẻ thường được khen ngợi bởi sự nỗ lực của mình thì vui vẻ, thậm chí rất yêu thích những vấn đề khó. Trước vấn đề khó, những đứa trẻ này thậm chí còn reo lên “Thật là thú vị, cháu thích những bài toán hóc búa kiểu này”. Và khi chúng thất bại, những đứa trẻ được khen là thông minh thường cảm thấy mất đi tự tin, trong khi những đứa trẻ được khen đã nỗ lực nói “có thể cháu đã chưa cố gắng hết sức mình”.

Dr. Dweck nói “Khen ngợi trí thông minh của trẻ không những không nâng cao sự tự tin của trẻ mà còn khuyến khích chúng quen với việc lo lắng sợ thất bại và sẽ lảng tránh không chấp nhận thử thách” “Tuy nhiên khi trẻ được dạy về gia trị của việc tập trung, tìm chiến lược và chăm chỉ hết sức mình khi đối mắt với những vấn đề khó trong học tập, chúng sẽ được khuyến khích giữ vững động lực, sự tự tin và thành thích của mình”

“Những đứa trẻ được khen ngợi là thông minh thường quan tâm đến việc xem những bạn khác làm như thế nào hơn là quan tâm đến việc tìm tòi chiến lược mới để giải quyết vấn đề”

Khen Con Như Thế Nào

Khen ngợi sự cẩn thận

“Con làm rất là tỉ mỉ, cẩn thận. Thảo nào mà mẹ thấy không vết mực nào rây ra ngoài cả”
“Sự tỉ mỉ cẩn thận của con đã đem lại kết quả tốt rồi, con thấy ko?”

Khen ngợi sự bình tĩnh

Mẹ rất vui vì con đã giữ được bình tĩnh mặc dù khối gỗ xây dựng của con cứ đổ xuống.
Hôm nay con đã dạy mẹ một bài học về sự cần thiết của việc giữ bình tĩnh đó.
Bình tĩnh luôn luôn giúp con đạt được mục đích đúng ko?

Khen ngợi sự nỗ lực, kiên trì, cố gắng

“cuối cùng thì con cũng làm được rồi”
“Mẹ rất hạnh phúc khi con biết nỗ lực như vậy”
“Con đã rất cố gắng kiên trì và con đã thành công rồi”

Khen ngợi sự chăm chỉ

“Con đã rất chăm chỉ ôn bài. Mẹ nghĩ con hoàn toàn xứng đáng với điểm 10 này”
“UI, hôm nay con mẹ chăm chỉ, hang say quá. Con thực sự làm mẹ tự hào về con.

Khen ngợi sự tập trung

“Mẹ để ý thấy con rất say sưa, tập trung xếp hình”. Có lẽ vì thế mà hôm nay con xếp tiến bộ hơn hẳn”
“Mẹ để ý thấy con đã rất tập trung đọc sách. Chắc con đã thấy rất nhiều điều thú vị trong câu chuyện đó”

Mỗi lần trẻ thành công trong bất cứ việc gì, đặc biệt những việc đòi hỏi sự cố gắng, nỗ lực, cha mẹ hãy áp dụng những câu khen ngợi này. Sau một thời gian, hãy để ý sự thay đổi trong cách ứng xử của con trước thất bại hay những vấn đề khó và chờ đợi cảm giác hạnh phúc khi nghe con hào hứng khoe thành quả của mình và nói “Mẹ ơi, con làm đi làm lại, làm mãi cũng được, con đã rất kiên trì mẹ ạ”.

Mỗi lời nói của cha mẹ đều có ảnh hưởng đến sự phát triển tính cách của con, tương lai của con

(Bài viết sưu tầm đâu đó trên mạng)

Transitional words (st)

Tue, 20 Aug 2013 00:00:00 +0000

Một số transitional words rất hữu dụng trong việc nối hoặc liên kết câu hoặc đoạn văn trong tiếng Anh. Để tránh sự lặp lại gây nhàm chán, các transitional words tương đương nên được dùng xen kẽ.

Addition

also, besides, furthermore, in addition, moreover, again

Consequence

accordingly, as a result, consequently, hence, otherwise, so then, therefore, thus, there upon

Summarizing

after all, all in all, all things considered, briefly, by and large, in any case, in any event, in brief, in conclusion, on the whole, in short, in summary, in the final analysis, in the long run, on balance, to sum up, to summarize, finally

Generalizing

as a rule, as usual, for the most part, generally, generally speaking, ordinarily, usually

Restatement

in essence, in other words, namely, that is, that is to say, in short, in brief, to put it differently

Contrast and Comparison

contrast, by the same token, conversely, instead, likewise, on one hand, on the other hand, on the contrary, rather, similarly, yet, but, however, still, nevertheless, in contrast

Sequence

at first, first of all, to begin with, in the first place, at the same time, for now, for the time being, the next step, in time, in turn, later on, meanwhile, next, then, soon, the meantime, later, while, earlier, simultaneously, afterward, in conclusion

Diversion

by the way, incidentally

Illustration

for example, for instance, for one thing

Similarity

likewise, similar, moreover

Direction

here, there, over there, beyond, nearly, opposite, under, above, to the left, to the right, in the distance

Sắp xếp trích dẫn (citations) trong (La)TeX

Thu, 15 Jan 2009 00:00:00 +0000

LaTeX cho phép tạo các trích dẫn (citations) để tham chiếu đến các tài liệu tham khảo tương ứng. Các trích dẫn, tùy theo định dạng (format/style) của tài liệu, có thể dạng số như [1, 2] hoặc dạng chữ như [Lamport94].

Một ví dụ đơn giản như sau

\LaTeX{} \cite{latex} is a document preparation system for the \TeX{} \cite{tex} typesetting program.

sẽ tạo ra kết quả

LaTeX [1] is a document preparation system for the TeX [2] typesetting program.

Trong ví dụ trên, [1] và [2] là các trích dẫn. Tài liệu tham khảo tương ứng sẽ được để trong phần \bibliography{}. Nhiều trích dẫn liên kế có thể được gom lại và liệt kê chung trong một cặp dấu ngoặc vuông như sau [6, 4, 5, 22, 27, 29].

Mặc định trong trường hợp nhiều trích dẫn, LaTeX không xếp thứ tự và không thu gọn cho dễ nhìn. Nếu muốn thu gọn, ví dụ như [4–6, 22, 27, 29] thì có thể dùng natbib hoặc cite.

Hãy đặt những dòng sau ở phần đầu file .tex, trước \begin{document}.

\usepackage{cite}

hoặc

\usepackage[sort&compress]{natbib}

Sau đó, chạy latex/pdflatex và bibtex như bình thường. Dạng trích dẫn trong bài sẽ được sắp xếp và nhóm lại hợp lý và dễ nhìn hơn. Gói natbib có một đặc điểm là tương thích tốt với gói hyperref (dùng để tạo các liên kết trong tài liệu pdf). Do đó, nếu cần dùng hyperref thì bạn nên sử dụng natbib.

Alex T.

Bringing Maven and AWS CodeArtifact Together

AWS CodeArtifact Repositories

Apache Maven Settings

CodeArtifact Authentication

Problems & Solutions

Maven CLI Wrapper

Updating Maven Settings

Happy Coding 💪

References

Building Node Canvas Container for AWS Lambda

Chromium and macOS Environment Variables

Method 1: Using Login Items

Method 2: Via LaunchAgents

Conclusion

Lenovo Thinkpad T430s Supervisor Password Removal

Disassemble T430s

Clear Supervisor Password

P.S.

References

Making JSP work with Spring Boot 2

Creating a Spring Boot Project

Project Configurations

Dependencies

Auto-configurations

View Resolvers

Embedded Servlet Container

Spring Boot + JSP Login Page

Login View

Login Controller

Handling GET Requests

Handling POST Requests

Handling Log-Out

Data Binding

Data Validation

Building and Executing

Updates

References

Tour de Spring Boot: DI meets CoC

A Quick Start

Project Bootstrapping

Build and Dependency Management

Autoconfiguration

App Launcher Configuration

Properties Loading

Components and Beans Configuration

Web Configuration

Data Access Configuration

Development Support

Testing

Troubleshooting

Developer Tools

Finale

Spring Boot properties issues with trailing whitespaces

The fun …

, the problem …

, the root causes? …

… and the solutions?!

Back to basics: Test-driven Spring JDBC

Background

A simple CRUD project with Spring JDBC

Defining Domain Entities

Configuring Database Connection

Test-Driven CRUD

CRUD Tests

CRUD Implementation

Handy macOS QuickLook configurations

QuickLook Settings

Forcing reloading the list of generators

Showing the list of generators

Resetting QuickLook caches

QuickLook Plugins

QLStephen

QLColorCode

QLCommonMark

QuickLook JSON

QuickLook CSV

HetimaZip

Suspicious Package

qlImageSize

Lessening `compinit` invocations

Optimising `path_helper`