I have been doing some serious research on the public exposure of Elasticsearch and Kibana instances recently, and here are a few of the reports I made, found to be valid. I’ve been enjoying it so much:

But my enjoyment must be someone else’s pain. Literally thousands of Elasticsearch and Kibana instances are publicly exposed to the Internet. They are just up for grabs, and that is a really serious problem. Sometimes, even top renowned tech companies make mistakes about this, exposing their internal data exposed to the public. The nature of the data stored in these instances are sometimes quite sensitive and are never meant to be shared: be it phone numbers, email addresses, home addresses, names, credit card numbers, even (plaintext!) passwords, internal API logs, private user data specific to the service, and more and more. I’ve seen millions of such data through this journey, which I never expected.

I honestly feel so sad for the users who have their data exposed to the public and probably don’t even know about that because it’s the developers who make mistakes, not them. I saw data from huge game studios, shopping malls, and even some of the biggest banks. It’s awful and I wanted to help fix that.

But the staggering number of instances and indices make it is somewhat difficult to find and collect interesting, or ‘useful’ data from them, which may lead to being awarded of bug bounties. To quickly gather data and filter useful and interesting ones, I decided to craft a tool that would automate removing lot of garbage data that nobody is interested in and help discover interesting data instead.

So here it is: elasticpwn.

elasticpwn: a brief overview

In this article, I will walk you through the general workflow of using elasticpwn. For the detailed explanation and full options, see README.

Prerequisites: docker-compose, go^1.17, Shodan or Binaryedge account (free account available for both)

1. Install the tool

go install github.com/9oelM/elasticpwn/elasticpwn@latest

2. Get a list of URLs of publicly exposed Elasticsearch and Kibana instances from OSINT platforms like shodan.io or binaryedge.io. Here are some starter dorks:

binaryedge: find publicly exposed elasticsearch instances in the US

elasticSearch AND country:"US" AND product:"Elasticsearch REST API"

shodan: find publicly exposed elasticsearch instances, excluding false positives

elasticsearch 200 -"adminer_sid" -"MiniUPnPd"

3. Process the urls so that they are in the form of something like:

123.123.123.123:80
124.124.124.124:443

.. and so on.

and save them in a text file.

4. Launch a local mongo server (you don’t have to use mongodb, but for the sake of this tutorial, I will just use it — you can also use json output option)

curl https://raw.githubusercontent.com/9oelM/elasticpwn/main/docker-compose-mongo-only.yml -o docker-compose-mongo-only.yml && docker-compose -f docker-compose-mongo-only.yml up -d

5. Run elasticpwn to collect data from the instances. This will collect and analyse all possible data from the instances with multiple threads.

elasticpwn elasticsearch -f list_of_elasticsearch_instances.txt -murl mongodb://root:example@172.17.0.1:27017/ -of mongo -t 12

6. Generate a report. This command will generate a directory called /report under CWD. There will be a lot of data, like really a lot, and you can’t just review them all unless there is some form of review system. So there goes the reason for the existence of report feature. By default, elasticpwn will do its best to find interesting data like email, password, api logs, etc using its built-in regex matching and show it in addition to basic information about an instance. Based on this data, you can skip uninteresting ones and only leave interesting ones.

elasticpwn report generate -cn elasticsearch -murl mongodb://root:example@172.17.0.1:27017/

7. View the report

elasticpwn report view -d ./path-to-report -p 9999

Here’s a sample report page containing information about a single Elasticsearch endpoint:

elasticpwn: persisting your research

elasticpwn also offers a way to store and retrieve the reviewed URLs, so that you don't have to review the same URLs again with the same report page. It often becomes very confusing because some instances are quite similar and the number of the instances is just too large.

All you need is a persistent mongo database and an additional installation of elasticpwn-backend.

1. Generate the report with -dn option:

# -dn should be the url where elasticpwn-backend will be hosted 
elasticpwn report generate -cn elasticsearch -murl mongodb://root:example@172.17.0.1:27017/ -dn http://localhost:9292

2. Install elasticpwn-backend:

go install github.com/9oelM/elasticpwn/report/elasticpwn-backend@latest

3. Launch it. You need to specify your persistent mongodb URI as mongodbUri option:

elasticpwn-backend -mongodbUri=mongodb+srv://username:pw@somewhere.mongodb.net/default-collection-name?retryWrites=true&w=majority -databaseCollectionName=elasticsearch_reviewed_urls -databaseName elasticpwn -port 9292

2021/12/28 05:38:54 Connected to database
2021/12/28 05:38:54 Inserting
2021/12/28 05:38:54 Test url already created before
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env:   export GIN_MODE=release
- using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /ping                     --> main.PingHandler (4 handlers)
[GIN-debug] GET    /urls                     --> main.GetUrlsHandlerGenerator.func1 (4 handlers)
[GIN-debug] POST   /urls                     --> main.PostUrlsHandlerGenerator.func1 (4 handlers)
[GIN-debug] DELETE /urls                     --> main.DeleteUrlsHandlerGenerator.func1 (4 handlers)
[GIN-debug] Listening and serving HTTP on 0.0.0.0:9292

4. With the server turned on, view the report (probably in another shell):

elasticpwn report view -d ./path-to-report -p 9999

Then, the report should be available, and each time you hit the ‘review’ button either with the keyboard shortcut or a click, the URL will be saved in the database for you to come back later or filter out duplicates later. If you are consistently generating report — say per week — then persistent database probably the answer.

Securing your Elastic stack

These are some really minimal steps to get your Elastic stack secured:

• set xpack.security.enabled as true if you are running an old version of the Elastic stack. One really good news is that this is going to be true by default in the newest versions. This will help protect lots of instances from being exposed to the world:

Currently, security features are disabled when operating on a basic or trial license when xpack.security.enabled has not been explicitly set to true. This behavior is now deprecated. In version 8.0.0, security features will be enabled by default for all licenses, unless explicitly disabled (by setting xpack.security.enabled to false).

• set password: bin/elasticsearch-setup-passwords interactive
• use WWW-Authenticate header to restrict access
• put the instances inside the private network and use VPN
• and so much more…

Elastic has a detailed documentation on security, so make sure you check it out if you are the guy who manages the Elastic stack in the company.

Conclusion

So far, we have looked at:

• the problem of data exposure from numerous Elasticsearch and Kibana instances
• how elasticpwn helps detect and analyse the exposed data
• simple ways to secure the Elastic stack

Thank you, and your feedback is always appreciated.

Originally published at https://9oelm.github.io/2022-01-02--elasticpwn-how-to-find-and-collect-data-from-exposed-elasticsearch-and-kibana-instances/.

This is a full guide to setup, develop and deploy a backend app with a recently released container image feature for lambda on AWS.

Needless to say, if you are a great fan of Docker, you would know how amazing it is. What you test on local is what you get when you deploy it, at least at the container level.

Since this feature is quite new, there have been lots of rabbit holes I fell into, and I’m sure others will too, so I will break every piece of rabbit hole down so that nobody gets trapped into it. This guide starts from the real basics like making a user or setting up terraform, so feel free to skip to the section you need.

Reason to use Terraform and SAM CLI together

Well, it seems that Terraform supports building a Docker image and deploying it to ECR out of the box, but after lots of digging, I noticed that things would get simpler if I just build docker image in another pipeline and deploy it with a few lines of shell script. So Terraform will used to define resources excluding the build and deployment process. There’s no problem with that.

And, what SAM CLI? Terraform cannot replace SAM CLI and vice versa. SAM cli is useful in developing local lambdas because it automatically configures endpoints for each lambda and greatly removes barriers to the initial setup. Since lambda functions are ‘special’ in the way that they only get ‘booted up and called’ when they are invoked (unlike EC2 or Fargate), just doing ts-node my-lambda.ts would not make it work. Of course there are many other solutions to this matter (such as sls) but in this guide I will just use SAM CLI. But for many reasons SAM makes me want to use other better solutions if any... The reason follows right below.

Disclaimer for the people who are looking for how to ‘hot-reload’ docker container for typescript or javascript based lambda: it won’t work smoothly as of now. The best bet is to use nodemon to watch a certain directory to trigger sam build every single time, and in another shell launch sam local start-api. It works as expected, but the current problem I see from here is that every single time it sam builds, it would make another Docker image and another and so on, so there will be many useless dangling images stacked up in your drive, which you will need to delete manually because SAM CLI does not support passing in a argument that's equivalent to docker run --rm. Anyways that's the story, so this is the reason I might want to try some other solutions. More on this on the relevant issue on Github. Please let me know if any of you had a good experience with sls because I haven't used it much yet.

Ok. Now let’s write some code.

Setup AWS for Terraform

First, make sure that you’ve installed and authorized on your AWS CLI. Installing AWS CLI is kind of out of scope here, so please follow the guide on AWS.

After successful installation, run:

You will be prompted to enter Access Key ID and Secret Access Key. Depending on your situation, there are different ways of how you can handle this, but for the sake of simplicity we can make one user (from AWS console. You will probably only use it for ‘Programmatic access’) that would have these policies for applying Terraform codes.

This one for setting S3 bucket as a backend:

And this one for locking the state:

And the next one is quite tricky; because we will temporarily enable permissions related to managing IAM because we will first need to make a role from which we could assumeRole whenever we try to plan and apply our IaC.

For now we can just go onto AWS console and make this policy:

Make sure you will need to narrow down to specific actions and resources used after everything is done (for production usage).

Now, now that you’ve made three distinct policies (or all in one, depending on your preferences), attach them to the user that you’ve just crated for running aws configure.

Setup Terraform

If you haven’t already, install terraform by following an instruction from the official website. Just download the binary and move it to the bin folder.

Now verify version of terraform

And then make main.tf file in your project directory (I personally put it into IaC folder because there will another folder for the 'real' .ts codes for the backend):

Now, run terraform init:

Then, we will need to add s3 backend and state locking. But before then, make a table on Dynamodb and also a bucket on S3, each for hosting IaC backend and locking the state.

Now we will need to add more to the policy on DynamoDB we created because we want to create a table:

Then you could write this code (by the way, it may be a good idea to put this below IaC in a different general-purpose repository because the current repository is meant to be only used for lambda-related resouces. But for the sake of this article I will just write it away here):

And then, also add S3 backend (you will need to add relevant IAM policies too here, but since we know how to do it, I will cut the explanation):

Now, run terraform apply, verify the changes, and enter yes. DynamoDB table and S3 Bucket should have been created. Here's the code so far:

Now, add s3 backend and state lock:

We are also going to use Docker provider, so add that too:

Now because you’ve added a backend and another provider, we will need to run terraform init again, and then terraform apply. Run it.

Setting up lambda

Now we will need to develop lambda on the local machine. Install SAM CLI:

Note that the outdated versions would not support running Docker containers, so make sure that your version is the latest.

Now, we won’t run sam --init, because it will make it difficult to make the server into a monorepo structure. The reason that we will want to make it into a monorepo is that it will make it much easier to propery dockerize every single lambda and deploy it with dependencies that each of them only require to have. Instead we will use lerna to initialize the server folder.

And as usual:

Then it will give you this layout:

Then, add your first function package. For the sake of this example, let’s assume that we want to make a REST API, composed of many lambdas, each returning ‘hello’ as a response in different languages (which is totally useless in reality, but at least useful here). Our first lambda will be an English one.

Now the directory structure will look like this:

Now, under server, we will need to add some utils to build and invoke the function locally. Add modify the server/package.json as follows and of course, run npm i again:

To add some explanation to what we are trying to do: these devDependencies are going to be package-wide dependencies. These are not specific to any one of the functions that we are going to build; They will help in tooling general stuffs. That's what we put them here.

Dependencies:

• @types/node: we will need this to give proper type definitions for 'built-in node' modules like fs or path.
• concurrently: just a script runner.
• lerna: you know it.
• nodemon: this will help us watch a directory and build Docker image again.

Scripts:

Now, you need to create template.yml for SAM cli to consume and run what we want to run.

We won’t be able to run sam build or sam local start-api yet, because we still need to setup Dockerfile and ECR repository.

So far we have added template.yml for running SAM CLI:

Now we will add Dockerfile in packages/hello/.

This will be the content for Dockerfile:

To go through it line by line:

• amazon/aws-lambda-nodejs:14 is the official amazon image for lambda. Current LTS of nodejs is 14, so we are using this. AS builder is related to multi-stage builds in Docker; it helps reduce the final Docker image size. Basically in this builder stage, we will only build the output to be included in the final image, and any dependencies installed in this step won't be included in the final output image.
• WORKDIR /usr/app: inside the docker image, set the working directory as /usr/app. There isn't any app folder in a normal docker image, so it will make app directory. We will put the compiled js code there.
• COPY package*.json tsconfig.json ./: we need these files for compiling typescript into javascriptt files.
• npm install: it will install dependencies.
• npm run build: it will compile typescript code into js.
• RUN ls -la # for debugging: it is merely for debugging. While building, docker will output what's inside there at that time, for you to verify if you are doing what you intended to do.
• FROM amazon/aws-lambda-nodejs:14: this is the second build stage in Docker. All outputs from the previous stage will be discarded in this stage unless explicitly specified to be included.
• RUN npm install --only=prod: it will only install dependencies but devDepdencies.
• COPY --from=builder /usr/app/lib /usr/app/lib: it explicitly refers to the previous builder stage to copy whatever that was inside /usr/app/lib to the current /usr/app/lib. In this case, it will copy all compiled javascript code.
• CMD [ "/usr/app/lib/index.handler" ]: the command should be path-to-lambda-handler-without-extension.handler. That's just how it works.

Now we’ve added a Dockerfile. Now let’s setup basic environment for lambda:

You will need to modify tsconfig to use modern javascript features; Most prominently, add the following. This will allow you to use Promise API. I recommend turning other options too, especially those related to strict-type checking:

Modify packages/hello/package.json too. Be noted that any dependencies you add to be included in the final, compiled output code (javascript) will need to be added to dependencies, not devDependencies:

Now, add a really simple lambda:

So far, we have created these:

Now, create nodemon.json under server/ to watch and build files:

After creating nodemon.json, you can start running npm run watch or npm start. It would do two things: build Dockerfile as you make any changes under packages/ directory, and host a local endpoint for the lambda. It will be similar to hot-reload although it seems more like a hack; you won't need to cancel and run sam local start-api again once you make a change. If it does not work, try again after creating ECR first.

Oh, and you can delete __tests__ and lib/hello.js because we are not using them. Anyways, now we are kind of ready to build this function into a docker image. Let's try it:

Everything’s cool, docker build succeeded. You can try running the image and test the request:

This is where SAM CLI should start to come in. But before then, we will need to make a ECR repository with terraform. Let’s go back to terraform for a while.

Back to Terraform: assuming role and ECR

Now, we will need to create a role first because we will relay on that role to get required permissions to create whatever resource we want to. This is called ‘assuming a role’, and the reason why it’s deemed to be a good practice is that you won’t have to create multiple credentials (probably multiple users) to do certain thing that requires permissions. Instead, you borrow the permission for the period of time when you plan and apply the changes in the resources.

So how do we do it? First, let’s create hello_role.tf:

For this article, we won’t be diving deep into specific policies, so we will just allow almost all resources without specifying them in detail. For real-world usage, you will have to define exact statements giving just the right permissions.

What we are doing here, essentially, is that we are allowing localtf user to assume the role of hello_role that possesses all policies to run the hello server stack. This is called 'creating a trust relationship' (you will see this if you do this process on AWS). This way, localtf won't always have to hold all permissions it needs. It only acquires them only when needed (i.e. deploying)

Once you are done writing hello_role.tf, run terraform apply to make changes.

Now, go back to main.tf and add:

Once you add assume_role, now you can create any resources you want, using the permissions given by that role. Let's now make an ECR repository. Make ecr.tf:

Run apply and terraform will soon make ECR.

Building and pushing the image to ECR

Now that we’ve made an ECR, we can go back to our server and write a little script to login, build and push the image of the lambda we are writing.

It’s pretty much straightforward; just get your AWS cli ready for using; and authenticate to be able to use ECR from Docker CLI:

Next, tag your Docker image as latest and separate timestamp at the time of the build, so that latest tag will always be the latest built image, and then another image will remain for the record, which you can use later to revert back or do other things in some cases.

Now, you can test it out yourself as such:

After this, you will be able to see on AWS ECR:

as you can see, the images are going to be tagged by timestamp, and the latest built image will be always tagged as latest, and you are going to reference this tag in Terraform to apply newly built Docker image to lambda.

So far, we’ve made changes like so:

Now the majority of the prepartion is done, so we can move onto creating actual lambda and API gateway.

First, the most important part: you want to create lambda itself from Docker.

There is already a module just made for this, so use it to make lambda. Once you are done, apply the changes.

The key here is that you are going to reference an image URI from ECR where the tag is the latest. If you have previously built and pushed a new docker image, the hash of the image is going to be different, thus causing a redeployment of the lambda function. Otherwise it has no idea if the docker image is new or not.

Again, after running the change, you can see the lambda on AWS console:

as you can see, compared to other lambdas, it has the package type of ‘Image’, which means it’s not from a Zip, but a Docker image.

You should be able to see the image URI (including the hash) of the image at the bottom of the information about lambda. If you click on that image URI, you will be navigated to the latest image on ECR that you just built and pushed.

Now, you will be able to test lambda on AWS Lambda console, but what we want in the end is something like sending GET /hello to a certain domain and receiving a response. To be able to do that, we need to setup API Gateway.

For this example, we will setup a domain at api.hello.com.

Here’s how:

I will explain the code one by one.

aws_api_gateway_rest_api will create a REST api. It usually does not include a single endpoint; it usually contains multiple, like: api.hello.co/hello, api.hello.co/bonjour, api.hello.co/nihao, and so on. On AWS, it is equivalent to a single row in APIs tab:

aws_api_gateway_resource: to put it very simply, you can think of this as a single API endpoint that has not yet been deployed. In this case we create a single endpoint ending with /hello.

In the below example, we have created two different resources with OPTIONS and POST, to the same path (hidden by black overlay). We will talk about creating OPTIONS resource to handle preflight requests later. For now, it would suffice to know that creating a REST resource means creating a certain endpoint.

aws_api_gateway_method: Now, you want to create a REST method for that resource. Our /hello endpoint will just require no auth (out of scope of this article), and be a GET method.

aws_lambda_permission: By default, there's no permission for API gateway to invoke lambda function. So we are just granting a permission to it so that it can be executed.

aws_api_gateway_integration: API gateway supports transforming (filtering, preprocessing, ...) a request before it reaches client or a response from the client before it reaches the actual lambda. We are not doing any special things here for this example, but you may want to use it in the future. For more, read relevant AWS docs.

aws_api_gateway_stage: API gateway supports separating API into different stages out of the box. You should use this to separate your API across production, staging and development environments. For now, we will only make a stage for current terraform workspace, which is assumed to be dev across all examples in this article. Once you apply your changes, you are going to be able to see this on your AWS console:

aws_api_gateway_deployment: This is equivalent to clicking on 'Deploy API' on AWS console.

Once resources are created in API gateway, they must be deployed in order to be reachable from external clients. One little problematic thing is redeployment; Even if you make a change to your REST API resources, it will not get deployed if redeployment argument does not change. There are mainly two ways of getting around this:

1. Use timestamp() to trigger redeployment for every single apply. Using this approach, lambda may be down for few seconds while redeployment. But it is for sure that it always deploys, so I would just go with this one if my service does not handle many users.
2. Use md5(file("api_gateway.tf")) to trigger redeployment whenever this file changes. But you need to always make sure that EVERYTHING related to API Gateway deployment only stays inside this file.

Ok. So far we have set up lambda and basic API Gateway configurations. Right now, you can test your API on Postman like this: first, go to AWS API Gateway console, and find a specific endpoint that is deployed to a certain stage. There should be an ‘Invoke URL’ at the top of the page.

Now, open up Postman, and

1. Insert your invoke URL
2. Click ‘Authorization’, and choose the type ‘AWS Signature’, and enter AccessKey and SecretKey. These keys should be coming from one of user's credentials from AWS IAM Console. If you do not have one specialized for calling an API set up with lambda and API gateway from local environment, make one user for that and get the keys.
3. Insert your AWS Region.
4. If your API requests any more query parameters or body, insert them.
5. Click send, then it should work.

If you do not provide AWS Credentials in your request header, it won’t work, because so far your API can be only used by IAM users known to the API gateway, and if you are just sending a request from your local computer without providing any access and secret keys, it won’t know it’s you. To make it work even without providing credentials for any public APIs intended to be exposed to client applications, you should now configure custom domain.

So far we have made changes to make lambda and API Gateway resources. The list of files we should have by now is the following.

Next, we will see how to create a custom domain and relate that domain to the REST API we just made.

Creating Terraform resources for custom domain

Now, the problem is that we have the API, but it’s not callable from any external client applications, which is a common case for many projects. So we want to register a domain first to represent our endpoints.

Before making a change for custom domain, we need to setup another AWS provider because we will need to use us-east-1 region for Edge-optimized custom domain name ( that's the only region that supports creating an ACM certificate for Edge-optimized custom domain name).

There are two choices available for an API endpoint: 1. edge; 2. regional. If your endpoint should be accessed by worldwide clients, use edge; if your endpoint is specifically confined to be used in one specific region in the world, use regional. If you don’t know what to do, it totally safe to go with edge for now. First, add another aws provider:

Then, you write the actual code to make a certificate and custom domain.

Make sure you get the existing hosted zone ID (or any other zone ID that you intend to use) from AWS Route53 Console:

Use that ID to create Route53 Record for the custom domain name.

After you apply your change, you will be able to see ACM certificate being created on AWS Certificate Manager:

Just make sure you verified the status to be ‘Issued’ and the validation status to be ‘success’. You may need to wait for several minutes before this completes. Also, if your certificate is not showing up, make sure that you are on us-east-1, not anywhere else.

After you are done with this, now you can go back to API Gateway again, and configure custom domain. Now that you’ve registered a domain, you can see it right up from API Gateway console:

In the certificate dropdown, you should be able to see the domain that you have just created. Do not make create domain name there on the console. Now come back to terraform and let’s write the equivalent code for that.

You are just going to fill out the options that you just saw from the API Gateway console. Just fill out the relevant info about certificate, domain name, and endpoint config.

Now, this is important: you need to create another Route53 Record to map your custom domain to cloudfront. Once you create your custom domain, AWS creates ‘API Gateway domain name’, circled with red in the below picture:

You need to route the traffic to api.hello.com to this API Gateway domain name (an example of an API Gateway domain name would be asdfasdfasdf.cloudfront.net as long as you are using EDGE). That's what we are doing with aws_route53_record.custom_domain_to_cloudfront. Otherwise, the response to your API will keep showing some weird errors that are really hard to guess the causes of. I found AWS really lacking a documentation on this part, so please be advised on this one. You need to create another Route53 Record.

You will be able to verify by entering Route53 console and looking for api.hello.com. It should appear as the following:

After that, you don’t have to do many things; just add base path mapping resource, in api_gateway.tf. Even if you do not have an additional trailing path to your endpoint, you must create a base path mapping. Otherwise your API won't be exposed to the public.

After applying this change, verify that your API mapping has been created:

Now, you can go back to Postman, and test your api by requesting GET api.hello.com/hello. What may be confusing here is that you are not adding any path in base path mapping. If you add hello as a path, your API endpoint will be configured as api.hello.com/hello/hello, which is obviously not what we want. So do not add any path mapping if you already have configured your path in aws_api_gateway_resource). Anyways, request and response to the API endpoint should work as expected if everything has been setup correctly so far.

Enabling OPTIONS (preflight request)

Now, our client application, of course, is not Postman, so usually clients will request OPTIONS api.hello.com/hello first, and then request GET api.hello.com/hello, if they intend to send CORS requests, which is a very common case (read more about that from MDN docs)

If you have not done anything related to handling OPTIONS request, it’s very likely that you will get some error in your client application, like this:

So let’s do it! There’s already a handy module written by a great dev, so we will just use that:

Note that if you have any custom headers, you must define it in your config. Next, verify that OPTIONS requests are now allowed on the console:

Also, you will need to change your lambda’s response header too. Just adding Access-Control-Allow-Origin": "*" will work:

Now, apply the changes, and go back to your client app and retry the request. It should be working.

Summing up

If you have followed everything, you will have created these files:

So far, we have looked at how to setup, develop and deploy a dockerized lambda application with Typescript, Terraform and SAM CLI. There are tonnes of things to cover on lambda.. maybe next time, it will be on using resources inside VPC from lambda. I hope you enjoyed this and found some valuable insights. Thank you.

Originally published at https://9oelm.github.io/2021-03-13-complete-end-to-end-guide-for-developing-dockerized-lambda-with-typescript-terraform-and-SAM-cli/

Complete End-to-end Guide for Developing Dockerized Lambda in Typescript, Terraform and SAM CLI was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

Disclaimer: We will focus on useSelector itself in this article, rather than third-party libraries like reselect, because it's out of the scope of the article.

How was it like before?

Now, before talking about useSelector, we need to know some background. Let's go back to pre-function component era, where all we needed to care about where the props. Not so much you needed to do. Choices were PureComponent and shouldComponentUpdate. Nothing else. Like this:

Right. So if you make such component, this component is only going to update when shouldShowRed is changed. Otherwise, it is going to stay still even if its parent renders for some reason. It goes the same for shouldComponentUpdate; It just gives you additional tooling to specify your own method of telling the component when to update.

Using it with Redux?

Again, using the component with redux was pretty straightforward too. Just create a container and pass states and dispatches (I didn’t write any mapDispatchToProps for the sake of simplicity) mapped as props:

Right. Let’s assume that we have RootReduxState as we have seen from the code above. And this is just going to work so well. Nothing difficult here. SomeDiv will keep being efficient at its best because PureComponent is working well. But how should we precisely port this example to a function component?

Function component

Well, it’s easy. Just use useSelector, Right?

Perfect. But what if you need more than just shouldShowRed? The moment you start to get more than one thing from useSelector (which is a very normal case), you are going to need additional optimization efforts with appropriate knowledge.

If we were to use shouldShowGreen too, we could choose:

Option 1

Or, we could:

Option 2

Alternatively:

Option 3

Or, just a small variant..

Option 4

useSelector forces re-renders

Which method have you been using? Whichever one you are using, you should be able to give reasons! Now, before having a look at each option, let’s go back to useSelector and see what it does.

So let me bring the most important thing to the table immediately: useSelector forces re-render of your component. It's not a prop, but it can still make your component re-render. Really? Yes. Let's look at the offical documentation:

When an action is dispatched, useSelector() will do a reference comparison of the previous selector result value and the current result value. If they are different, the component will be forced to re-render. If they are the same, the component will not re-render.

So how do you judge if they are different? By running strict equality (===) comparision. ( Click here if you are feeling like looking at the official repo's source code). In the source code, useSelector by default uses a function called refEquality, and all it does is simply const refEquality = (a, b) => a === b.

This means that if you are returning an object from useSelector for whatever reason, useSelector will cause a re-render every time an action is dispatched from the store, simply because objects of the same structure (same keys and values) do not strictly equal each other in javascript. For example, { a: 1 } === { a: 1 } is false. Official doc says the same:

returning a new object every time will always force a re-render by default.

Do they really force it? Yes. From the source code:

So… now, with this in our mind, let’s go back to the options we saw previously.

Option 1

Catches:

1. You are just writing a pair of shouldShowRed and shouldShowGreen for three times just to take the desired state out.
2. This will cause a re-render forcefully every time an action is dispatched, because you are returning a new object from your selector.

Verdict: not good enough.

Option 2

Catches: you are returning the entire state from your selector, and destructuring it outside of the selector. This will too cause a re-render. What’s the point of having the selector callback if you intend to receive the entire state? This is a bad practice. It’s just tantamount to passing the entire state in mapStateToProps. You don't do that there. So, why here?

Verdict: not good enough.

Option 3

Catches:

1. you are calling useSelector twice, and this does not matter. According to the official doc:

Because of the React update batching behavior used in React Redux v7, a dispatched action that causes multiple useSelector()s in the same component to return new values should only result in a single re-render.

2. strict equality is functioning as properly for each selected state because you are returning a primitive from each selector.

Verdict: usable.

Option 4

Catches: It’s just the same as option 1 or 2. It will cause a re-render too.

Verdict: Not good enough.

Improving the bad options

Thankfully, redux gives us a chance to insert our own equality functions. The concept is the same shouldComponentUpdate or the equality callback in React.memo. We could do this:

or,

Something like that. However you should really note that using deepEqual cannot ever be fast if you are trying to equal a large object (Dan Abramov said it, too!):

Deep equality checks are generally a bad idea. If the inputs are shallow enough that you think deep equality would still be fast, consider just using [JSON.stringify(obj)] instead. Note it won't compare functions.

Also "fast-deep-equal" is such a misleading library name. If it's deep it can't claim to be "fast" because you're at the mercy of your data structure. Some seemingly innocent changes to it can make the comparison super slow.

Note also that you only want to run deepEqual on what you need. In the code snippet above, you are also comparing shouldShowBlue which is a part of RootReduxState['conditions']. But you don't need that anyways, but you are still comparing it. Make sure you select and compare what you only need.

But why would you try to optimize it in the first place?

Well, at first, it’s totally okay. Your app has ~100 components only, your redux state is quite shallow, and it does not take a long time to render whatever’s being rendered.

The problem comes at two points:

1. once you start to scale your application. If you succeed in making a popular application, you are going to support more features, and thus, need more components. Your components will be numerous, leading to the point where re-render of expensive components will be causing a sluggish interaction.
2. once you start to care about users using low-end devices. You want to support users with low-end spec computers. You want to support mobile devices with inherently less performance than most computers. Just get a 6x slowdown on your CPU from Chrome’s performance tool and try to see how long it takes for your components to react.

Conclusion

So far we looked at possible problems with using useSelector and how to solve them:

1. If you are returning an object from your selector callback, it’s going to force re-render by default.
2. To prevent re-render, either let your selector return a primitive type, or use a custom equality function.

That’s it. Thank you!

Originally published at https://9oelm.github.io/2020-09-13--How-to-make-useSelector-not-a-disaster/.

How to Make useSelector Not a Disaster was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Yes, Chrome’s performance tab is overwhelming

First time you look into all the charts, you have no idea what means what. What do those different colors even mean? It’s not self-explanatory at all. There needs to be a guidebook.

In this article,

1. I will briefly walk you through all functionalities of Chrome’s performance tab, with majority of the usual curiosities resolved.
2. Using what we learned, we will find what made our React components so slow and think about how to improve them.

A very simple example with React

I wrote some short components, <Item /> and <LongList />, to make an example for our article.

And it’s showing me this, which is full of 4500 rows of <Item /> s and a button to make all 4500 rows again:

The entire loading is quite slow if you slow down CPU on Chrome, like this:

This means that on lower-end PCs, it’s likely that it’s going to be slow like that too. Let’s guess for a second. What’s making this so slow? Network requests? generateManyRandomThings because it's producing so many elements in an array? Save your guess for now. We are going to investigate this thorougly as we are reaching the end of this guide.

You are able to see the entire source code on Github (Chrome Performance profiles included) and access the deployed website on netlify. Feel free to open up performance devtools on Chrome and try on your own (the production deployment might behave differently, though)

An overview of all major functionalities in a single slide

So here’s what those colors and jargons mean. Check this image out. I wanted to have it in a single slide so that I could grasp everything at one go (the resolution of the image below is poor, please open here for a good resolution).

Breaking down the overwhelming parts

I will explain each part that seems to be overwhelming or not self-explanatory, one by one.

Reload button

Clicking on the reload button will reload the page and start profiling automatically, and profiling will stop 3 seconds after the load event. This is useful for performance monitoring for initial load.

Disable javascript samples

Disabled (checked):

Enabled (unchecked):

Clicking on this option disables showing call stacks of Javascript functions in the recording of Main panel. This means you cannot see names of functions called during execution at all.

Enable advanced paint instrumentation

After ticking this checkbox, record the performance again.

1. Click on the frame from Frames section;
2. Click on Layers panel at the bottom. You are going to see the layers at the time of that frame and can inspect them.

1. Click on any of paint events from Main section
2. Click on paint profiler at the bottom. Inspect instructions and time taken to paint.

Timeline

Summary panel at the bottom of your Performance tab will show colors, and they directly match to the middle CPU region. Each same color means the same thing in the Timeline and Summary panel.

If you look at the rightmost side of your timeline, you are going to see FPS (topmost), CPU (middle), and NET (bottommost).

The little red section in the pink bar means that FPS dropped so siginificantly that it may have harmed UX.

You can observe from the above picture that the FPS graph (green part) shows a drastic decrease in that pink and red region.

The part with hatched fill pattern means that the work was done off-main-thread (I do not know why Chrome does not tell us this in any direct way, although it’s quite confusing for people who are new to performance monitoring).

Timings

React records time taken for mount and update on Chrome by using User Timing API.

You can see mount and update timings for all components. If you click on the chart, the bottom panel on Performance tab will show you detailed information about that component and its neighbor components. We will talk about this later more.

If you want to inspect how and what functions are used in a specific components, we will have to dig Main panel. Hold on a sec until then.

Chrome also records important events:

• DCL (DOMContentLoaded): HTML is fully loaded and DOM tree is constructed. Stylesheets, images, iframes, other external resources may not have been loaded.
• L(Load): Basically, DCL + all external resources loaded.
• FP (First paint): ‘the time between navigation and when the browser renders the first pixels to the screen, rendering anything that is visually different from what was on the screen prior to navigation’
• FCP (First Contentful Paint): time when some contentful thing (text, image, canvas, or SVG) is painted for the first time). The very first time user can start consuming page content.
• FMP (First Meaningful Paint): the time it takes for a page’s primary content (biggest layout change, probably the main HTML structure, and web fonts) to appear on the screen.
• LCP (Largest Contentful Paint, not in the picture below): the render time of the largest image or text block visible within the viewport.

Experience

Very recently, Chrome introduced ‘Experience’ panel. It reports layout shifts for now. A layout shift is an unexpected movement of page content that usually happens because resources are loaded asynchronously or DOM elements get dynamically added to the page above existing content, harming UX.

I tried this on medium.com and luckily, I got one Layout shift:

Notice that you could click on some little text buttons on the right-bottom side at about the beginning of 1600ms. But as you reach the beginning of 1800ms, these texts suddenly go away (probably to the bottom).

Chrome defines this behaviour as a source of harmful UX, especially when it comprises some crucial activities like payments or credentials.

You can inspect more information from the panel at the bottom:

Main

So this is the interactive view of call stacks. The functions at the top are the ones that were called earlier; the ones at bottom called later. You can click around and inspect using any of Summary, Bottom-Up, Call Tree, and Event Log tabs from the bottom. These are the most important parts I wanted to know:

• Colors: they don’t have special meaning, but charts are colored in the way that would be easy for you to see pattern of execution.
• Long tasks: the API reports any tasks that execute for longer than 50 milliseconds (ms). Keeping every task under 50ms ensures visible, immediate response for user once some interactive action was taken by user.

If you look closely enough, you will find that Chrome already colors the duration of time that should be reduced to make a long task into a ‘bearable’ task, with red-gray hatch pattern.

• Navigation (pro tip): move around with WASD, and arrow up and down key for your ease. It’s unlikely you will use a mouse to navigate around Main once you get comfortable with this.

Raster and compositor

Rasterizing is converting visual information into pixels on the screen. Uses multiple threads (which are not part of the main thread).

Compositing is a separating parts of a page into layers (happens in a thread that’s different from rasterizing thread), and rasterize them separately (also happens in other threads).

We will cover these two in a different article because these make such a long topic themselves. For now, let us just be aware that there are these two things.

Summary, Bottom-Up, Call Tree, and Event Log

Summary

The range represents the duration selected in the timeline.

The legend on the right side shows:

• Loading: time taken for loading resources
• Scripting: time taken to parse and evaluate javascript
• Rendering(Layout): time taken to computing styles and positions of elements on the page. Not sure why Chrome names it Layout in the Main panel and Rendering here.
• Painting: time taken to fill in the pixels
• System: effectively, can be also called as others. All activities not belonging to the categories Loading, Scripting, Rendering, Painting and GPU.

Bottom-Up

• Self Time: the time taken to execute code in the function itself, excluding the time to execute other functions it calls. (in the example, useState and moutState have almost 0% self time because the long lines of the chart are suddenly ending at generateManyRandomStrings which is causing all the heavy works.
• Total Time: the time taken to run functions (up to the end) from the current function.
• Look for high total time branches, and then find the blocks with high self time. That’s the one that might be causing some performance problems.
• In the example above, generateManyRandomThings accounts for 1.3% of rendering time of LongList, cumulatively. But its total time is 22.5%. So it should be that the functions it's running are doing costly works (the actual works done purely inside that function only aren't really slow).

• Don’t be confused: if you open consecutive dropdowns under the first tab, you are tracing back to the initial function that was called. You are going down in the call stack. You are NOT going up.

Call Tree

• It traces back to the root activity(or activities) that caused the function to be called.
• In the example above, the root activity of initBackend is Function Call.

Event log

• It shows chronological order of events.
• In the example above, LongList was first called at 256.7ms, and then useState at 290.0ms, and then generateManyRandomThings at 290.2ms.

So how do I use this with React?

Good question! Let’s now go back to our code. Then, let us slow down our CPU again. I’ve set it as 4x slowdown. Now, let’s try to feel the first loading speed (click on ‘Start profiling and reload page’ button). For your information, you can download the performance profiles I’m using in this example from github.

Well, for sure, it took some time until you could first see anything from the browser. Now let’s look at the record:

First let’s find from when the browser was fully ready before it started doing something with React.

We can see from above that the browser spent the first 500ms destroying the previous webpage (which was actually the same localhost:3000), requesting over network, parsing, and compiling resources like javascript and html files.

Right after 500ms, we see that actual functions from React library itself as well as our ./src/index.js are being called. Let's look at it with Timings tab:

The red rectangle is the execution of LongList (components are functions too!), and the blue rectangle is the execution of many Item s.

My personal, naive guess (without knowing anything well about how React should work in this situation) on why the FCP was so late before looking into the performance log was that generateManyRandomThings is running very slowly, simply because I thought it's putting so many elements into an array after numerous calculations.

But after inspecting the log, we find that the culprit is not generateManyRandomThings (under LongList):

1. Even just by looking at the chart, it’s evident that generateManyRandomThings is just a small part of the entire render process of <LongList />. generateManyRandomThings was called twice because useState was repeatedly called twice during render; but both were still short. Below is the duration of the first execution of generateManyRandomThings. As you can see, it's really just a small part (red circle) under render:

2. render took almost 2 secs total time, while generateManyRandomThings took only 54.1ms total time (2.7% of render's total time):

Then what’s the main cause of slowness? Well, it turns out the culprit is <Item />, as you might have easily spotted already.

There are so many repeated processes going on for each of 4500 different <Item />s. How can we optimize this then? In reality, we are perhaps going to use something like a virtual list, so that you only render what's inside your current viewport only. That's going to reduce a lot. Because making a virtual list is not the main topic here, let's just pretend we made it by reducing items down to 100 items, and see how our React app does again. For the sake of proof of performance, let's still make 4500 elements in the array and only render 100 <Item /> s, like this:

And the result…? Without even inspecting deeply, we can already feel that the initial load (the instant when the rows changed) time was greatly improved.

ah, now we can see that the self-time of LongList finally became somewhat similar to total of execution of all Item s:

Although it might sound very simple that mounting 4500 components is slower than putting strings and numbers into an array, it’s always worth trying out auditing performance using Chrome’s Performance tab, because until you inspect the real performance with logs and stats, you are essentially just guessing which part is supposed to be the slowest. In complex React apps, there’s only a low chance it’s going to work well that way.

Last curiosity

But why is useState being called twice anyways in a single mount, like this?:

Well, it was not related to something we did! Dan Abramov says it always happens when you use <React.StrictMode>. So there's nothing wrong with it. If we change

to

It’s going to show you only a single execution of useState, like this:

Closing notes

So.. that’s it! We covered most basic concepts. In near future, I will also cover GPU, Interactions, and some few parts I saved for another chance. We will also look into how to optimize complex React + Redux applications with Chrome’s Performance panel later.

Thank you for reading!

Originally published at https://9oelm.github.io/2020-08-03--Learn-all-major%20functionalities-on-Chromes-Performance-tab-and-practice-it-with-a-simple-React-project/.

Learn All Major Functionalities on Chrome’s Performance Tab and Practice It With a Simple React… was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

I was struggling at my company trying to write some tests for redux-observable operations. Most of the network requests were managed by RxJS, and none of them were covered by tests. I wanted to write tests but just did not know how to.

But I really just wanted to start from the basics — I mean, the very basics, including why, and how we might want to use redux-observable.

tl;dr

If you want to hop straight into the source code, here it is.

What is reactive programming?

Reactive programming is programming with asynchronous data streams.

• Typical events are async event streams and you want to observe over them
• You have useful functions to combine/create/filter those streams

A stream is a sequence of ongoing events ordered in time. It can emit three different things: a value (of some type), an error, or a “completed” signal.

We capture these emitted events only asynchronously, by defining a function that will execute when a value is emitted, another function when an error is emitted, and another function when ‘completed’ is emitted.

So this is essentially going back to the observer design pattern.

Why bother to use RxJS?

Abstraction

If you use RP, you don’t have to really worry about implementation details, because it gives you a high level abstraction layer. So you never have to bother with Promises and await in javascript anymore. Just declaratively implement what you want to do, and that's it.

Async works made easier

Nowadays async operations have become a common thing because you might need to handle UI interactions & tonnes of network requests.

So there’s a good chance that it is going to be a good fit for heavily network based application (both for frontend and backend).

Lots of helper functions to simplify work declaratively

You have dedicated helper functions to implement what you want to do, which otherwise you might do in a very complex way. Even if this does not fully come to your mind right now, you are going to notice it as you go through this article.

Why bother to use it with Redux?

Now we have a brief understanding of advantages of using RxJS. Now, we want to handle actions in Redux, but not all actions are synchronous.

For a simple example, I have written an application that fetches text from the server and shows it, like this:

I assumed that you have a prior knowledge of some Typescript and React fundamentals including hooks.

This is the project structure (omitted unimportant ones):

Redux-related things first

To be able to understand true benefits of using redux-observable, we need to see how redux works first, so that we can compare (you can skip this part if you already have an idea of how you would handle async actions with bare redux)

In redux, as you know, we mainly have actions and reducers. I have created three types of actions:

1. StartRequestText: I will dispatch this right before fetching from the dummy text API.
2. FinishRequestText: I will dispatch this right after receiving the text.
3. ErrorRequestText: I will dispatch this as soon as I encounter an error in the process described above.

The above actions are implemented like this in actions.ts:

And I simply defined some action types in constants.ts:

Putting them into reducer.ts is not difficult at all. We just have to update the state for each different action. You can see that I'm updating isLoading, text, and errorMsg individually according to each type of an action:

reducer.ts

Again, not so much in store.ts. Just combine reducers (although we just have one, which is not usual in production envrionments) and configure redux devtools extension, and create store.

store.ts

And I defined the types we need in types.ts:

App.tsx In the parent component, we just want to show two things: the button and the text.

DummyTextRequestButton.tsx

In this component, we have the logic of dispatching all of the actions we have defined earlier. You have to recognize that we want to port the logic in handleClick to redux-observable later, because right now it's not reactive at all.

DummyTextViewer.tsx

This component consumes the redux state. Shows error, text, or loading based on the state.

With RxJS and redux-observable?

First of all, we are going to introduce relevant modules to our project:

Then we want to get rid of somewhat complex logic inside DummyTextRequestButton.tsx:

We can do it like above because we are going to port our logic to redux-observable.

Then, we are going to create something called epics.ts inside redux folder:

An epic is really just something that listens to a redux action and outputs other actions accordingly.

So how is that done in the code above? We can see that startRequestTextEpic is listening to an action of the type called C.START_REQUEST_TEXT only. So this code, won't get executed it any other actions are dispatched from redux.

Ok, forget about mergeMap for now and then let's go on for now. We can see that we are using RxJS's own fetch method called fromFetch, which essentially just makes fetch into an observable.

Now you process the response from fetch, and the text inside the response would be a payload to finishRequestText action.

Otherwise, if you have an error while fetching, the error is going to be caught and will dispatch errorRequestText action.

In this way, we have successfully ported our implementation in handleClick to inside an epic, and all that's left for we us to do is actually configure a few things in store.ts to be able to use redux-observable in our project:

So, after adding redux-observable, now we have some benefits:

1. Reactiveness. You can easily handle asynchronous actions and state updates in redux by reacting to actions being dispatched.
2. Separation of concerns. Now you don’t have to care about what to do after startRequestText in DummyTextButton, because all of the logics have been ported to startRequestTextEpic.
3. Great toolchains. Right now we have used just few functions from RxJS, but combination of many functions will make it easier to process your data streams.

Extending further by utilizing what RxJS has

Now we have one little problem: when a user clicks on the button many times in such a short time, the request is going to be sent more than one time, which might be kind of useless, as shown in this gif:

I clicked on the button three times, but actually what you all need is just a single request. So here’s what you can do:

We just added debounce, and it is going to filter out only one action of type START_REQUEST_TEXT for a second, which means even if you click on the button three times in a single second, the request is going to be sent only once, like in the gif below:

Alternatively, what you would want better is actually switchMap; It would cancel any pending request and switch to the last one:

Now, let’s say that you would have another button to show some other text, like this:

Because we now have an additional UI, we need to handle one special case (we are not going to cover other cases, although there are many): you click on request dummy text button, but you change your mind to see hi instead. Then you would need to cancel and finish your action.

For the sake of simplicity, I’m not going to show the entire code but epics.ts to demontrate that (you can still look into the entire codebase from the Github repository):

We now have two major parts: sendRequest$ and cancelRequest$. In cancelRequest$ ovservable, we are going to force finish requesting text, because there is no need to request the text anymore if you have switched to see the hi text.

race would take the job of cancelling out any other ongoing action if the other action is finished. It's just like Promise.race in javascript! So, let's see this in gif:

Checkpoint

Do you now get why you would use redux-observable? It equips you with a nice set of tools to perform side-effects after actions are dispatched in redux. Otherwise it should have been a great pain for you manage events like cancellation manually, in vanilla javascript.

But hey, we are not done yet. Let’s finish it with a guide to how to test these epics.

Testing your epics

Method 1: Test output actions based on input actions.

Whoa, lots of things to go through, right? But basically, all we are doing are just:

1. Mocking functions from certain modules to make them work according to our purpose. We are mocking rxjs.race and FF.fromFetch because, for race, we don't want race won't work well in a test environment and we can decide on which observable to emit, instead of depending on the code itself, and for fromFetch, we don't want to use real fetch because it is going to take more time for no useful reason.
2. Running the epic with initial action and state. The epic will run, and spit out another set of actions as a result.
3. Testing the equality of the output actions with the expected. We just need to test if the epic gives expected set of actions.

That’s pretty much it! But we need to test more, so why not simplify this process with a simple function?:

Now, we can turn out test into:

Much less boilerplate, right?

Now we can write out tests for other things as well, like this:

Now you’ve got to see all of your tests pass:

Method 2: Use marble tests

Previously, we’ve only used some basic methods to test epics. Using marble tests would allow you to focus more on when and what things happen.

If you are not familiar with the marble diagrams, please refer to ReactiveX’s explanation on marble diagrams. Alternatively, there are many other good resources out on the web.

For the marble syntax, refer to rxjs API docs.

Hot vs Cold observables

There was a great article on Medium that simplified the definition of hot and cold observables, so I excerpted the definitions from it. (Or look at rxjs API doc)

cold: begins subscription when the test begins.

cold(--a--b--|, { a: 'Hello', b: 'World' }) → Emit 'Hello' at 30ms and 'World' at 60ms, complete at 90ms.

hot: begins subscription at the point of caret.

hot(--^--a--b--|, { a: 'Hello', b: 'World' }) → Subscription begins at point of caret, then emit 'Hello' at 30ms and 'World' at 60ms, complete at 90ms.

Testing events in relation to time

What we couldn’t really answer using method 1 was something like ‘can actions be really be cancelled if there are multiple action inputs, because we are using switchMap?'

Now because we have a control over time, this is totally possible.

This one is really basic; We request text, and for the sake of the test, we delay the response from fromFetch by 1ms. Then we run START_REQUEST_TEXT, and expect the output action to be FINISH_REQUEST_TEXT after another 1 ms elapses. Simple. Right?

But what about some more complex test, like actions being cancelled because of switchMap?:

In the above code, we are faking fromFetch again, and this time it is going to get you a response after 5ms. Now you decide to START_REQUEST_TEXT four times just in 4ms, in a row, each for each 1ms.

Then what is supposed to happen is that all of a, b, c get cancelled, because they are cancelled by the actions dispatched right after them, at the time fromFetch has not returned a response yet. This is what switchMap exactly does. It will cancel other observables and only emit the most decently projected observable, which is d (after 5ms).

So that’s basically it! Of course, we could again make an abstraction of this testing logic, just like we did for runEpicTest, but too bad - I'm too tired right now, so I will leave that work left for you. :)

Conclusion

So far, we’ve covered:

1. why you might want to use rxjs and redux-observable (maybe you could suggest more reasons in the comments section)
2. how you could replace redux based code into a redux-observable one
3. extensibility (benefits) of using redux-observable, especially for a complex logic
4. two methods of how you can test your epics

Appendix

Just some last words for you.

Suspicion on ‘always a pass’

Sometimes, it’s hard to know if your test is really working or just being passed because jest finishes tests earlier than the time you want. Make sure you use done, and if you are still not sure, try to break the test by changing something like expect(...).toEqual(...) to expect(...).not.toEqual(...). Then you should see something like this:

if your tests are still being passed, then there’s a problem.

Shoudn’t I test how epic changes the state as well?

No. Because you are supposed to test an epic on how set of actions outputs another set of actions.

You are going to test that in your tests for reducer, because reducer is actually where the action will change the state.

What epic really does is just until dispatching actions. Of course, it’s related to the state because it uses it, but it does not directly modify the state. That’s going to be left for the role of the reducer.

Source code repository

• redux implementation
• redux-observable implementation and tests

And… thank you for reading!

Originally published at https://9oelm.github.io/2020-01-24--Fundamental-yet-extensive-introduction-to-why-and-how-you-might-want-to-use-redux-observable-for-async-actions/.

Extensive introduction to why and how you might want to use and test redux-observable was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

The first encounter with the problem.

I was coding as usual. And I faced an odd encounter with how forEach works. Here goes the code to give an example of that. Imagine the code below is the code from one of the libraries I was using:

And I spotted the code in my company’s app that was kind of doing:

Of course, I did not want this. It’s against DRY principle. So I refactored the code to:

Cooler and more succinct.

But I was very well tricked into thinking that this would just work. See what this code gave:

Oops. Everybody’s dead. Ok. Now you are starting to get a grasp of why.

It’s the binding. Let’s check what this is actually doing inside our code:

Well, no surprise. It outputs:

Ok. So here’s the main point of this article:

Just passing in the reference of a function that uses this referring to somewhere else than a globalThis, into a forEach might cause an error in javascript because this will point to a global this.

So what do we do? Here are some things to let you know:

1. Explicitly call the function

Yeah this works. This will save some of the guys’ lives.

Same for the normal function as well:

2. Pass thisArg as an argument

These are the parameters for forEach:

You can put in the object to be pointed to as this inside saveFromZombies.

3. Use bind

This is an explicit binding. You tell the javascript engine that you want saveFromZombies to be bound to walkingDead object.

Further..

Of course, we can, and should, apply the same principle when dealing with map, filter, ... and more.

Summary

• We have looked at how this might lose context when we put a reference of a function as a callback to forEach.
• The solutions are: (1) Explicitly call the function, (2) Pass thisArg as an argument, and (3) Use bind.

Happy forEach coding!

Originally published at https://9oelm.github.io/2019-10-12--[...].forEach(sayHello)-does-not-always-say-hello/

[…].forEach(saveFromZombies) does not always save people was originally published in JavaScript in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

We had a bunch of large component trees in our company’s web application. And I saw them re-rendering from the top to deep down to the bottom for no reason. At first, I thought it would be easy to get them correct.

Of course, the tree was much bigger than this. The tree didn’t just reach the end. This made useless re-rendering even a bigger concern. From the top to bottom, Provider, ConnectedRouter, App, PersistGate, ComponentA, ComponentB, .... and so on.

And I saw that there were no use of shouldComponentUpdate/ PureComponent/ memo in our app yet. So I decided to make use of them, still not yet aware of what was to come for me.

Takeaway 1: Parent component pretty much does not care about its children

I was optimizing the app and witnessed an interesting fact. Here’s the code to demonstrate it:

And do you think Child1 and Child2 are going to re-render upon Parent component's mouseEnter? See what happens:

Oops. They re-render. But I do not see a reason why (The same thing happens for class components, if you wonder. You can try that!). The children do not even have their own props or states to listen for any changes. How can this happen? This is from React's own documentation:

shouldComponentUpdate() is invoked before rendering when new props or state are being received. Defaults to true.

It defaults to true! What… Ok. React is made that way.

Then how can we make it better?

Takeaway 2: use the children prop

Ok. The only difference is that we use children prop.

And the result?

It does not re-render the children components anymore! I still have not understood exactly why React can in this code but cannot in the previous code, but this is what it is.

Takeaway 3: use React.memo

memo literally memoizes the functional component receiving a certain props, and if the props stay the same, it will just return the previously rendered (same) component. The result? Same as solution 1. It does not re-render Child1 and Child2. So I'm just gonna copy and paste the gif from above for those who are lazy to read what's written, just like me:

Ok. But if you are using passing in nested objects as props, you should implement your own areEqual function for memo, like this:

Yeah. I know it’s an ol’ school style to check nested object equality, but it works well. In practice, you use something like isEqual from react-fast-compare that would handle nested object comparison for you.

Ok. Now, what’s the result?

Child1 no longer knows obj prop stays the same because memo only runs a shallow comparison between prevProps and nextProps But Child2 exactly knows, by using areEqual function, that obj is actually staying the same and therefore it does not need to re-render itself.

Takeaway 4: Use PureComponent/shouldComponentUpdate

Of course, you might wanna use your class component. So here’s a brief explanation on that as well:

Above is an exact replica of the functional component implementations. Here are the details:

1. You use PureComponent for Child1, but it won't serve its purpose, just like memo, because it has a nested object as its prop. It will keep re-rendering upon its parent component re-rendering. It would work if there were no nested object properties.
2. You use shouldComponentUpdate for Child2 and it will work. It compares nested objects just like areEqual. In case there is a state, you can use nextState to compare with this.state (previous state).
3. It’s just so annoying to write lots of boilerplate of codes for a class component. That's why I prefer functional components... (yeah I digressed a bit)

Of course, the result is the same as the last code snippet. Let’s again copy and paste the same gif for the people:

Takeaway 5: Deal with functions from props when comparing

There are many cases where props are functions, especially when you plug your functions into components with redux’s mapDispatchToProps, or passing your functions as props from parent down to children to get notified of what happened in children.

Dealing with functions might sound like really a basic thing, but it can really harm your application’s performance if you don’t do.

We see lots of patterns like this:

Details:

1. Now, the obj prop no longer an object. It's just become num.
2. You are passing a function called sayHiInChildren to Child2 as a prop.
3. You shallowly compare if all props are the same, both in Child1 and Child2, albeit using different ways (Child1 is not using a custom areEqual function, but Child2 is. But essentially they are doing the same thing in this context).
4. Everything is going to re-render although props to Child1 and Child2 are staying exactly the same. Look at the result:

But why!? Well..

In javascript, there is hardly a way to compare functions.

This will always return false, because functions are objects in javascript, meaning they are compared not with values, but references:

The only way to make it possible is to have the same reference:

Then how can we deal with functions in props? Well, there are 4 ways:

1. Exclude them
2. Stringify them
3. Use useCallback
4. Bind the function to the class

1. Exclude them (not recommended)

You can use lodash’s _.isFunction to check if something is a function. Then you only have object properties that are not functions, to apply that to areEqual function.

Now, you can do this:

Result? Only parent component re-renders. Essentially, you are only comparing props which are not functions.

2. Stringify them (not recommended)

By default, JSON.stringify does not support stringifying functions. So you've gotta use third party libaries like jsonfn.

Let’s do this:

JSONfn.stringify will generate a string looking like:

Using this function will give you the same result as the last example. Here’s another copy and paste:

3. Use useCallback

Probably this is the wisest solution when you are trying to prevent rerender in a functional component.

const sayHiInChildren = useCallback(child => console.log(`hi~~~~~~~~~ — from ${child}`), [])

useCallback will look at the dependencies fed into the array and will only give a different function when some dependency changed. Otherwise it is going to return a memoized callback, which won’t cause a rerender even if it’s fed as a prop.

4. Bind the function to the class

As you know useCallback cannot be used in class components.What can we do in the class components?

You can either do:

class Parent extends React.Component {
  sayHiInChildren = child => console.log(`hi~~~~~~~~~ - from ${child}`);

  render() {
    return <div onMouseEnter={handleMouseEnter} style = {{border: '1px solid black', padding: '50px'}}>
      Parent. Count: {count}
      <Child1 num={1} sayHiInChildren={sayHiInChildren}/>
      <Child2 num={2} sayHiInChildren={sayHiInChildren}/>
    </div>
  }
}

or

class Parent extends React.Component {
  constructor(...args) {
    super(...args);
    this.sayHiInChildren = this.sayHiInChildren.bind(this);
  }

  sayHiInChildren(child){
    console.log(`hi~~~~~~~~~ - from ${child}`);
  }

  render() {
    return <div onMouseEnter={handleMouseEnter} style = {{border: '1px solid black', padding: '50px'}}>
      Parent. Count: {count}
      <Child1 num={1} sayHiInChildren={sayHiInChildren}/>
      <Child2 num={2} sayHiInChildren={sayHiInChildren}/>
    </div>
  }
}

The two methods differ only in the syntax (arrow function binds directly to the parent context, while traditional function has to be bound manually), and they bring the same effect. In this way, you can retain a consistent reference of a function, which will allow you to prevent useless re-render.

Summary

• If you do not use children prop, every component inside the render function of a parent will re-render by default.
• There are ways to prevent useless re-renders: (1) Use memo with areEqual, or (2) Use PureComponent or shouldComponentUpdate
• If there are functions in your props, before using areEqual or shouldComponentUpdate: (1) Exclude them (not recommended), (2) Stringify them (not recommended), or (3) use useCallback , or (4) bind the function to the class.

Originally published at https://9oelm.github.io/2019-10-02--Making-stupid-react-smart-in-re-rendering/.

Edit

First of all, thank you for the insightful comments.

1. The cost of stringifying the functions may be larger than the benefit of optimization itself. So don’t use stringify, really.
2. You should instead use useCallback in your functional component (see the comment section for an example). But still, I’m not sure how we could do that for a class component. Anybody got a good idea?

Yes, and you should really take into consideration how much tradeoff your optimization could bring. In other words, the optimization could not work as you expected.

Making stupid React smart in re-rendering was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.