ITBigQueryTest flakiness

[ajkannan]

Recently, these two tests have been failing more often than passing:

testInsertFromFile(com.google.gcloud.bigquery.it.ITBigQueryTest)  Time elapsed: 13.864 sec  <<< FAILURE!
java.lang.AssertionError: expected:<RECORD> but was:<PRIMITIVE>
    at com.google.gcloud.bigquery.it.ITBigQueryTest.testInsertFromFile(ITBigQueryTest.java:939)
testListAllTableData(com.google.gcloud.bigquery.it.ITBigQueryTest)  Time elapsed: 0.291 sec  <<< FAILURE!
java.lang.AssertionError: expected:<RECORD> but was:<PRIMITIVE>
    at com.google.gcloud.bigquery.it.ITBigQueryTest.testListAllTableData(ITBigQueryTest.java:640)

I will comment out these two tests for now, but we should uncomment and diagnose the problem when possible.

[aozarov]

/cc @mziccard

[mziccard]

This is a super weird error. What seems to be happening is that both a load job and a resumable upload fail to correctly insert rows from JSON. Rows are inserted but they're missing their record field.

The JSON we are inserting is:

{"TimestampField": "2014-08-19 07:41:35.220 -05:00","StringField": "stringValue","IntegerField": ["0", "1"],"BooleanField": "false","RecordField": {"TimestampField": "1969-07-20 20:18:04 UTC","StringField": null,"IntegerField": ["1","0"],"BooleanField": "true"}}
{"TimestampField": "2014-08-19 07:41:35.220 -05:00","StringField": "stringValue","IntegerField": ["0", "1"],"BooleanField": "false","RecordField": {"TimestampField": "1969-07-20 20:18:04 UTC","StringField": null,"IntegerField": ["1","0"],"BooleanField": "true"}}

Interestingly the load job succeeds:

"status": {
  "state": "DONE"
 }

And its statistics match the input file:

"statistics": {
  "creationTime": "1459547849376",
  "startTime": "1459547850058",
  "endTime": "1459547855974",
  "load": {
   "inputFiles": "1",
   "inputFileBytes": "525",
   "outputRows": "2",
   "outputBytes": "126"
  }
 }

Maybe @jcondit and @jtigani can help us on this.

[jcondit]

Can you provide table IDs and job IDs? How are you determining that the record field is absent (i.e., how are you reading the table)?

[mziccard]

We read the table with tabledata:list and it returns:

{
 "kind": "bigquery#tableDataList",
 "etag": "\"3lJH3XPQlFznBcrQRCytdsfdVIE/OcHGzn4HN1HVkVrJNbQnMuHcIn0\"",
 "totalRows": "2",
 "rows": [
  {
   "f": [
    {
     "v": "1.40845209522E9"
    },
    {
     "v": "stringValue"
    },
    {
     "v": [
      {
       "v": "0"
      },
      {
       "v": "1"
      }
     ]
    },
    {
     "v": "false"
    },
    {
     "v": null
    }
   ]
  },
  {
   "f": [
    {
     "v": "1.40845209522E9"
    },
    {
     "v": "stringValue"
    },
    {
     "v": [
      {
       "v": "0"
      },
      {
       "v": "1"
      }
     ]
    },
    {
     "v": "false"
    },
    {
     "v": null
    }
   ]
  }
 ]
}

as you can see record fields are null.

• Project: gcloud-devel
• Dataset: gcloud_test_dataset_temp_7f053b09_17fd_4aad_8b15_61f0948026d5
• Table: testing_table
• Job: job_Jyva8RqK7kwSxZlV1ewlbTe5J00

[mziccard]

Ehi @jcondit any update on this?

[mziccard]

When this issue was filed flaky tests were failing very often, today I managed to run them 10 times I had a success rate of 10/10. @jcondit did something change? Any feedback would be valuable.

[jcondit]

Apologies for dropping this earlier! I suspect this was https://code.google.com/p/google-bigquery/issues/detail?id=503, which was just fixed. (You might consider filing future issues over there or internally to reach a wider audience.)

[mziccard]

Thanks @jcondit happy to hear this is fixed now. I'll file it internally next time, just wanted to be super-sure before bothering you guys.