Create new object and specify primary key WITHOUT db query?

[mtitolo]

See edits below. My console gets flooded with these errors:

2014-08-20 12:00:29.939 app[29675:4407] Unknown error calling sqlite3_step (19: column postId is not unique) eu
2014-08-20 12:00:29.940 app[29675:4407] DB Query: INSERT INTO "Link" ("last_utc","author","kind","url","created_utc","name","lastUpdated","customImgURL","recommendedIds","gilded","hasCustomImg","selfText","thumbnail","edited_utc","score","elapsed","categoryClass","likes","commentListId","title","numComments","sortedCommentListId","postId") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)

I'm manually creating Links with alloc/init, then using EasyMapping to fill them with data from a JSON response.

NSMutableArray *models = [NSMutableArray arrayWithCapacity:JSONArray.count];
NSArray* ids = [JSONArray valueForKeyPath:@"data.id"];
NSDictionary* existingLinks = [Link keyedInstancesWithPrimaryKeyValues:ids];

for (NSDictionary *JSONDictionary in JSONArray){
    Link *model = nil;
    if ([existingLinks objectForKey:JSONDictionary[@"data"][@"id"]]) {
        model = [existingLinks objectForKey:JSONDictionary[@"data"][@"id"]];
    }
    else {
        model = [[Link alloc] init];
        model.postId = JSONDictionary[@"data"][@"id"];
    }

    [EKMapper fillObject:model fromExternalRepresentation:JSONDictionary[@"data"] withMapping:[self linktMapping]];

    [models addObject:model];
    [model save];
}

It does not appear that I can alloc/init and then specify the primary key. The PK I'm using comes from the server, and is guaranteed to be unique. It is not supposed to be generated on the client, so I cannot reliably override primaryKeyValueForNewInstance, which it looks like init calls.

So: how should I create new objects and specify a primary key without querying for them?

Edit: After more digging, the DB error comes from the fact that primaryKeyValueForNewInstance is somehow creating duplicate keys...

Edit 2: Switching to [Link instanceWithPrimaryKey:JSONDictionary[@"data"][@"id"]]; does not stop the flood of these error messages...

[marcboquet]

Is it possible that EKMapper modifies postId somehow?
The other code looks good if you switched to instanceWithPrimaryKey instead of alloc-init.

[mtitolo]

@apalancat It's likely that EKMapper will set the postId to the same value as JSONDictionary[@"data"][@"id"] as that's part of the object mapping. Setting the PK to the same value should not generate errors though...?

Also, if you see my second edit, instanceWithPrimaryKey: is also generating these errors. And, yes, this is the only spot Link objects are created and saved in to the DB.

[marcboquet]

Yeah that shouldn't generate errors, no. It's very similar to what I'm doing, the main difference is that I don't use an existingLinks array, all my instances are already loaded so I rely on FCModel's cache and call instanceWithPrimaryKey for every id (haven't tested the performance though, could be very bad).
But that shouldn't affect the outcome either...
I'm thinking maybe you have an "unusual" schema that's triggering an edge case? Like for example non-integer primary keys? Shouldn't matter but it's the only thing I can think of.

[mtitolo]

My primary keys are strings generated on the server and guaranteed to be unique. Any tips to debug exactly what it's doing? If I break inside a save block, I only see the proper data, including the postId from the server.

[marcboquet]

Based on the errors seems like it's doing an insert when should be doing an update?
Maybe the logic that determines it (line 873 in master) is failing somehow?

If that's the case, the existsInDatabase ivar is the one you want to track down, probably set in initWithFieldValues:existsInDatabaseAlready:

[mtitolo]

I added some logs. It's definitely doing a double-insert, instead of an update. The funny thing is, both times there's all of the data, including the primary key. The duplicate saves are milliseconds apart, so this might have to do with creating the object and then immediately updating it before it's done being inserted to the db?

2014-08-21 12:19:26.604 app[42981:5f0f] INSERTING WITH COLUMNS 
2014-08-21 12:19:26.605 app[42981:60b] INSERTING WITH COLUMNS 
2014-08-21 12:19:26.606 app[42981:3613] Unknown error calling sqlite3_step (19: column postId is not unique) eu
2014-08-21 12:19:26.607 app[42981:3613] DB Query: INSERT INTO "Link" ("last_utc","author","kind","url","created_utc","name","lastUpdated","customImgURL","recommendedIds","gilded","hasCustomImg","selfText","thumbnail","edited_utc","score","elapsed","categoryClass","likes","commentListId","title","numComments","sortedCommentListId","postId") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
2014-08-21 12:19:26.607 app[42981:3613] Unknown error finalizing or resetting statement (19: column postId is not unique)
2014-08-21 12:19:26.607 app[42981:3613] DB Query: INSERT INTO "Link" ("last_utc","author","kind","url","created_utc","name","lastUpdated","customImgURL","recommendedIds","gilded","hasCustomImg","selfText","thumbnail","edited_utc","score","elapsed","categoryClass","likes","commentListId","title","numComments","sortedCommentListId","postId") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)

[marcboquet]

It seems that the DB thread is 3613 but you have two other threads 5f0f and 60b where the "INSERTING WITH COLUMNS" code is called.
Is it possible that the problem is related to this? Depends on where the NSLog statement is but you could have a...

(dun-dun-duuun)

concurrency issue 😱

[mtitolo]

@apalancat Managed to track down the concurrency issue, which wasn't an issue until we rewrote our regex in c, I guess (it calls back to the main thread:60b within milliseconds now).

But this raises another question: is this framework purposely not doing any thread-safe kind of functions?

Also I was overriding save and checking for a non-FCModelSaveSucceeded result, and even though there was a DB error it still returned FCModelSaveSucceeded?

[marcboquet]

Usually concurrency is fine, since DB access is synchronous. But seems like you found an edge case. As far as I understand the issue is:

1. Two threads call save on the same object that's not in the database.
2. Both threads get to FCModel.m:850 roughly at the same time, so update is NO for both.
3. Two identical INSERT queries are added to the DB queue.
4. The second thread's insert fails (not sure why yours return FCModelSaveSucceeded, I haven't been able to reproduce this).

A solution could be to add a semaphore to save so the second thread would wait before deciding whether it's an insert or an update, but I'm not sure if that could introduce other problems.

@mtitolo Do you feel that's an accurate description of the issue?
@marcoarment do you think adding a semaphore to save would be safe?

[ahti]

If the problem is as @apalancat is describing, this should already be fixed by 7f64fa7, which moved the whole -save method onto the database queue.