Make output format of logger compatible with fluentd

[smacker]

Part of #123

Better make it configurable somehow. For example make a new environmental variable which can be TextFormatter, JsonFormatter, FluentdFormatter. And then additional variables to configure json formatter... But for now it looks like overkill.

[bzz]

Nice! What would be the simplest test plan for this?

Could you please also share it in here, if there is a simple way to verify this new format gets picked up by fluentd in k8s?

[smacker]

We have only one staging_aka_production environment.
So the only way to verify - deploy it there and raise some errors/warning (maybe we will need to create custom image with those errors)

[dpordomingo]

I'm confused...
I think we could avoid copy-pasting this bunch of code if we could use FluentdFormatter from joonix/log, as can be found at https://github.com/joonix/log/blob/master/fluentd.go

[smacker]

we don't have any policy about it.

@bzz what the party says?

This particular code can be found in 2 repositories:
https://github.com/joonix/log/blob/master/fluentd.go
https://github.com/elafarge/gin-http-logger/blob/master/logrus-formatters/fluentd-formatter.go
And I saw more which are absolutely the same but have different naming. For example:
https://github.com/everflow-io/logrus-gcloud-format/blob/master/formatter.go

and it's copy past of original logrus formatter + 9 lines of simple mapping code (can be written in 3):
https://github.com/sirupsen/logrus/blob/master/json_formatter.go

P.S. When I write Go I try to avoid 1 liners dependency hell as it is in npm, but it's up to project/company.

[smacker]

oops. My bad. Not 9 additional line. Just changes of 3 lines.

I got confused at first. Actually now it looks like it's possible to archive the same effect using only logrus. But I still need clarification about dependencies for the future @bzz

[dpordomingo]

idk... joonix/FluentdFormatter is what sirupsen/logrus suggests.
(or the fluentd hook that idk if it would suit our necessities)

[smacker]

you need fluentd hook if you don't use kubernetes.

Those only 3 lines formatter changes:

	data["time"] = entry.Time.Format(timestampFormat)
	data["message"] = entry.Message
	data["severity"] = entry.Level.String()

So we can just do:

&JSONFormatter{
  	FieldMap: FieldMap{
		 FieldKeyTime: "time",
		 FieldKeyLevel: "severity",
		 FieldKeyMsg: "message",
   },
}

[bzz]

The only requirements, in case of copy-pasting a code with some modifications, is to keep attribution (copyright) and make sure licenses are compatible.

@smacker was there any particular place where this code comes from?

Using logrus JSONFormatter with 3 lines of configuration sounds even better to me, as will avoid extra hassle with licenses/copyrights in future.

[smacker]

I believe I took it from some gist, then found out its actually fluend format not GCloud format. So I took name from joonix. And there is no license in files in logrus.

[dpordomingo]

LGTM

[dpordomingo]

could you ptal @bzz @carlosms

[dpordomingo]

@smacker I tested the tip of #181 in staging aka production and it worked as expected, so please rebase, squash and merge 🗡️

[bzz]

Looks awesome to me